Change URL Parameters For "Next" Page

[scrapefun]

When using Net:HTTP parser to scrape Google is it possible to change the URL parameters used for the second page?

So the first query would be normal first page Google query:
https://www.google.com/search?q=$query

But for the second/next page I want the query string to be:
https://www.google.com/search?q=$query&start=10&num=100


Thanks!

 

[Support]

Use $tools.query.add: http://en.a-parser.com/wiki/template-tools/#tools-query-adding-of-requests to add your own query.
In this case, it would look like this:

[% tools.query.add('https://www.google.com/search?q=' _ query _ '&start=10&num=100') %]

 

[scrapefun]

Where would I add that to this Task:

Spoiler: Task Screenshot

Spoiler: Task Import Code

eyJwcmVzZXQiOiJUd28gUGFnZSBUZXN0IiwidmFsdWUiOnsicHJlc2V0IjoiVHdv
IFBhZ2UgVGVzdCIsInBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsi
dHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0s
eyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1v
emlsbGEvNS4wIChXaW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUz
Ny4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNh
ZmFyaS81MzcuMzYifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicXVlcnlmb3Jt
YXQiLCJ2YWx1ZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZS5jby51ay9zZWFyY2g/cT1b
JSBxdWVyeSB8IHVyaSAlXSZwd3M9MCZnbD11cyJ9LHsidHlwZSI6Im92ZXJyaWRl
IiwiaWQiOiJwcm94eXJldHJpZXMiLCJ2YWx1ZSI6IjEwMCJ9LHsidHlwZSI6ImN1
c3RvbVJlc3VsdCIsInJlc3VsdCI6WyJwYWdlcyIsImRhdGEiXSwicmVnZXgiOiIo
Pzo8ZGl2IGNsYXNzPVwic3JnXCI+fDxkaXYgY2xhc3M9XCJnXCIgZGF0YS1odmVp
ZD1cIlxcZCtcIj4pKC4rPyk8IS0tbi0tPjxcXC9kaXY+PFxcL2Rpdj4iLCJyZWdl
eFR5cGUiOiJnIiwicmVzdWx0VHlwZSI6ImFycmF5IiwiYXJyYXlOYW1lIjoic3Jn
IiwicmVzdWx0cyI6WyJkYXRhIl19LHsidHlwZSI6ImN1c3RvbVJlc3VsdCIsInJl
c3VsdCI6WyJzcmciLCJkYXRhIl0sInJlZ2V4IjoiPGgzIGNsYXNzPVwiclwiPjxh
IGhyZWY9XCIoLis/KVwiLis/JyhcXGQrKScuKz8+KC4rPyk8XFwvYT4iLCJyZWdl
eFR5cGUiOiJnIiwicmVzdWx0VHlwZSI6ImFycmF5IiwiYXJyYXlOYW1lIjoic2Vy
cCIsInJlc3VsdHMiOlsibGluayIsInJhbmsiLCJ0aXRsZSJdfSx7InR5cGUiOiJv
cHRpb25zIiwiaWQiOiJjaGVja05leHRQYWdlIiwidmFsdWUiOiI8dGQgY2xhc3M9
XCJiIG5hdmVuZFwiPjxhIGNsYXNzPVwicG5cIiBocmVmPVwiKFteXCJdKylcIiBp
ZD1cInBubmV4dFwiIiwiYWRkaXRpb25hbCI6eyJjaGVja05leHRQYWdlTGltaXQi
OiIxIn19LHsidHlwZSI6ImN1c3RvbVJlc3VsdCIsInJlc3VsdCI6WyJwYWdlcyIs
ImRhdGEiXSwicmVnZXgiOiI8c3BhbiBjbGFzcz1cInNwZWxsXCI+U2hvd2luZyBy
ZXN1bHRzIGZvcjxcXC9zcGFuPiA8YS4rPz4oLis/KTxcXC9hPiIsInJlZ2V4VHlw
ZSI6ImkiLCJyZXN1bHRUeXBlIjoiYXJyYXkiLCJhcnJheU5hbWUiOiJtaXNzcGVs
bCIsInJlc3VsdHMiOlsia2V5Il19LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJw
cm94eWJhbm5lZGNsZWFudXAiLCJ2YWx1ZSI6IjAifSx7InR5cGUiOiJvdmVycmlk
ZSIsImlkIjoidGltZW91dCIsInZhbHVlIjoiMzAifV1dLCJyZXN1bHRzRm9ybWF0
IjoiWyUgSUYgcDEuaW5mby5zdWNjZXNzID09IDE7XG5cdEZPUkVBQ0ggcDEuc2Vy
cDtcblx0XHRtcCA9IChwMS5taXNzcGVsbC4wLmtleSA9PSAnJykgPyAnMCcgOiAn
MSc7XG5cdFx0cXVlcnkub3JpZyBfICc7ICcgXyBtcCBfICc7ICcgXyBsb29wLmNv
dW50IF8gJzsgJyBfIGxpbmsgXyBcIlxcblwiO1xuXHRFTkQ7XG5FTkQgJV0iLCJy
ZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6IlslIGxpbmVz
ID0gbGluZXMgKyBwMS5zZXJwLnNpemU7XG5VU0UgTWF0aDtcbidzZXJwcy8nIF8g
TWF0aC5pbnQobGluZXMgLyAxMDAwMDAwKSBfICcudHh0JyAlXSIsImFkZGl0aW9u
YWxGb3JtYXRzIjpbWyJbJSBJRiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMTtcblx0VVNF
IE1hdGg7XG5cdCdzZXJwX3Jhdy8xXycgXyBNYXRoLmludChxdWVyeS5udW0gLyAy
MDAwKSBfICcvJyBfIHF1ZXJ5Lm9yaWcgXyAnJztcbkVORCAlXSIsIiRwMS5wYWdl
cy4wLmRhdGEiXSxbIlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRVU0Ug
TWF0aDtcblx0J3NlcnBfcmF3LzFfJyBfIE1hdGguaW50KHF1ZXJ5Lm51bSAvIDIw
MDApIF8gJy8nIF8gcXVlcnkub3JpZyBfICcnO1xuRU5EICVdIiwiJHAxLnBhZ2Vz
LjEuZGF0YSJdLFsic2VycF9mYWlsL2ZhaWxlZC50eHQiLCJbJSBJRiBwMS5pbmZv
LnN1Y2Nlc3MgPT0gMDtcblx0cXVlcnkub3JpZyBfIFwiXFxuXCI7XG5FTkQgJV0i
XV0sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1ZXJ5Rm9ybWF0IjpbIiRxdWVyeSJd
LCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2Us
Iml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExldmVscyI6ZmFsc2UsInF1ZXJ5QnVp
bGRlcnNBZnRlckl0ZXJhdG9yIjpmYWxzZSwicXVlcnlCdWlsZGVyc09uQWxsTGV2
ZWxzIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7Im92ZXJ3cml0ZSI6ZmFsc2V9
LCJkb0xvZyI6Im5vIiwia2VlcFVuaXF1ZSI6Ik5vIiwibW9yZU9wdGlvbnMiOmZh
bHNlLCJyZXN1bHRzUHJlcGVuZCI6IiIsInJlc3VsdHNBcHBlbmQiOiIiLCJxdWVy
eUJ1aWxkZXJzIjpbXSwicmVzdWx0c0J1aWxkZXJzIjpbeyJzb3VyY2UiOlswLFsi
bWlzc3BlbGwiLCJrZXkiXV0sInR5cGUiOiJyZW1vdmVIdG1sIiwiYXJyYXkiOiJt
aXNzcGVsbCIsInRvIjoia2V5In1dLCJjb25maWdPdmVycmlkZXMiOltdLCJydW5U
YXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNGaWxl
IjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVDb25maWciOiJkZWZhdWx0IiwidG9v
bHNKUyI6IiJ9fQ==

 

[Support]

Just add it to your Result format.

 

[scrapefun]

I don't understand how to integrate it into my existing result format and do I still need to use the "Next Page" regex that I'm using or can I remove that since the $tools.query.add code seems to take care of the second query I need.

 

[Support]

In this case, the result format should look like this:

At the same time query template need to move from a Query format of preset into the General query format to avoid formatting the request obtained from $tools.query.add. Also need to remove Check next page, because this feature is already implementing $tools.query.add.

Spoiler: Code

eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs
InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy
aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl
cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX
aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws
IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi
fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi
OiIxMDAifSx7InR5cGUiOiJjdXN0b21SZXN1bHQiLCJyZXN1bHQiOlsicGFnZXMi
LCJkYXRhIl0sInJlZ2V4IjoiKD86PGRpdiBjbGFzcz1cInNyZ1wiPnw8ZGl2IGNs
YXNzPVwiZ1wiIGRhdGEtaHZlaWQ9XCJcXGQrXCI+KSguKz8pPCEtLW4tLT48XFwv
ZGl2PjxcXC9kaXY+IiwicmVnZXhUeXBlIjoiZyIsInJlc3VsdFR5cGUiOiJhcnJh
eSIsImFycmF5TmFtZSI6InNyZyIsInJlc3VsdHMiOlsiZGF0YSJdfSx7InR5cGUi
OiJjdXN0b21SZXN1bHQiLCJyZXN1bHQiOlsic3JnIiwiZGF0YSJdLCJyZWdleCI6
IjxoMyBjbGFzcz1cInJcIj48YSBocmVmPVwiKC4rPylcIi4rPycoXFxkKyknLis/
PiguKz8pPFxcL2E+IiwicmVnZXhUeXBlIjoiZyIsInJlc3VsdFR5cGUiOiJhcnJh
eSIsImFycmF5TmFtZSI6InNlcnAiLCJyZXN1bHRzIjpbImxpbmsiLCJyYW5rIiwi
dGl0bGUiXX0seyJ0eXBlIjoiY3VzdG9tUmVzdWx0IiwicmVzdWx0IjpbInBhZ2Vz
IiwiZGF0YSJdLCJyZWdleCI6IjxzcGFuIGNsYXNzPVwic3BlbGxcIj5TaG93aW5n
IHJlc3VsdHMgZm9yPFxcL3NwYW4+IDxhLis/PiguKz8pPFxcL2E+IiwicmVnZXhU
eXBlIjoiaSIsInJlc3VsdFR5cGUiOiJhcnJheSIsImFycmF5TmFtZSI6Im1pc3Nw
ZWxsIiwicmVzdWx0cyI6WyJrZXkiXX0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6
InByb3h5YmFubmVkY2xlYW51cCIsInZhbHVlIjoiMCJ9LHsidHlwZSI6Im92ZXJy
aWRlIiwiaWQiOiJ0aW1lb3V0IiwidmFsdWUiOiIzMCJ9XV0sInJlc3VsdHNGb3Jt
YXQiOiJbJSBJRiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMTtcblx0Rk9SRUFDSCBwMS5z
ZXJwO1xuXHRcdG1wID0gKHAxLm1pc3NwZWxsLjAua2V5ID09ICcnKSA/ICcwJyA6
ICcxJztcblx0XHRxdWVyeS5vcmlnIF8gJzsgJyBfIG1wIF8gJzsgJyBfIGxvb3Au
Y291bnQgXyAnOyAnIF8gbGluayBfIFwiXFxuXCI7XG5cdEVORDtcbkVORDtcbnEg
PSBxdWVyeS5vcmlnIHwgdXJpO1xudG9vbHMucXVlcnkuYWRkKCdodHRwczovL3d3
dy5nb29nbGUuY29tL3NlYXJjaD9xPScgXyBxIF8gJyZzdGFydD0xMCZudW09MTAw
JywgMSkgJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFt
ZSI6IlslIGxpbmVzID0gbGluZXMgKyBwMS5zZXJwLnNpemU7XG5VU0UgTWF0aDtc
bidzZXJwcy8nIF8gTWF0aC5pbnQobGluZXMgLyAxMDAwMDAwKSBfICcudHh0JyAl
XSIsImFkZGl0aW9uYWxGb3JtYXRzIjpbWyJbJSBJRiBwMS5pbmZvLnN1Y2Nlc3Mg
PT0gMTtcblx0VVNFIE1hdGg7XG5cdCdzZXJwX3Jhdy8xXycgXyBNYXRoLmludChx
dWVyeS5udW0gLyAyMDAwKSBfICcvJyBfIHF1ZXJ5Lm9yaWcgXyAnJztcbkVORCAl
XSIsIiRwMS5wYWdlcy4wLmRhdGEiXSxbIlslIElGIHAxLmluZm8uc3VjY2VzcyA9
PSAxO1xuXHRVU0UgTWF0aDtcblx0J3NlcnBfcmF3LzFfJyBfIE1hdGguaW50KHF1
ZXJ5Lm51bSAvIDIwMDApIF8gJy8nIF8gcXVlcnkub3JpZyBfICcnO1xuRU5EICVd
IiwiJHAxLnBhZ2VzLjEuZGF0YSJdLFsic2VycF9mYWlsL2ZhaWxlZC50eHQiLCJb
JSBJRiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMDtcblx0cXVlcnkub3JpZyBfIFwiXFxu
XCI7XG5FTkQgJV0iXV0sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1ZXJ5Rm9ybWF0
IjpbImh0dHBzOi8vd3d3Lmdvb2dsZS5jby51ay9zZWFyY2g/cT1bJSBxdWVyeSB8
IHVyaSAlXSZwd3M9MCZnbD11cyJdLCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZl
RmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExl
dmVscyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0ZXJhdG9yIjpmYWxzZSwi
cXVlcnlCdWlsZGVyc09uQWxsTGV2ZWxzIjpmYWxzZX0sInJlc3VsdHNPcHRpb25z
Ijp7Im92ZXJ3cml0ZSI6ZmFsc2V9LCJkb0xvZyI6Im5vIiwia2VlcFVuaXF1ZSI6
Ik5vIiwibW9yZU9wdGlvbnMiOmZhbHNlLCJyZXN1bHRzUHJlcGVuZCI6IiIsInJl
c3VsdHNBcHBlbmQiOiIiLCJxdWVyeUJ1aWxkZXJzIjpbXSwicmVzdWx0c0J1aWxk
ZXJzIjpbeyJzb3VyY2UiOlswLFsibWlzc3BlbGwiLCJrZXkiXV0sInR5cGUiOiJy
ZW1vdmVIdG1sIiwiYXJyYXkiOiJtaXNzcGVsbCIsInRvIjoia2V5In1dLCJjb25m
aWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVz
dWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVD
b25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9fQ==

 

[scrapefun]

Thanks! This works great.

I'm trying to setup a preset like the one I posted above except it only saves the raw html results and not the raw and parsed results.

I have it working but I can't figure out how to set it up without the 1st result file for the parsed results. I highlighted the result file I want to remove in the screenshot below. It does work as is but it would be great to completely remove the unnecessary result file entry.

When I remove that result file it doesn't work. I am guessing the problem is with the result format.

Spoiler: Preset Code

eyJwcmVzZXQiOiJyb3VnaDMiLCJ2YWx1ZSI6eyJwcmVzZXQiOiJyb3VnaDMiLCJw
YXJzZXJzIjpbWyJOZXQ6OkhUVFAiLCJkZWZhdWx0Iix7InR5cGUiOiJvdmVycmlk
ZSIsImlkIjoiZ29vZENvZGUiLCJ2YWx1ZSI6WzIwMF19LHsidHlwZSI6Im92ZXJy
aWRlIiwiaWQiOiJ1c2VyLWFnZW50IiwidmFsdWUiOiJNb3ppbGxhLzUuMCAoV2lu
ZG93cyBOVCA2LjE7IFdPVzY0KSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBs
aWtlIEdlY2tvKSBDaHJvbWUvNDcuMC4yNTI2LjExMSBTYWZhcmkvNTM3LjM2In0s
eyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InByb3h5cmV0cmllcyIsInZhbHVlIjoi
MTAwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InByb3h5YmFubmVkY2xlYW51
cCIsInZhbHVlIjoiMCJ9LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJ0aW1lb3V0
IiwidmFsdWUiOiIzMCJ9XV0sInJlc3VsdHNGb3JtYXQiOiJbJSBJRiBwMS5pbmZv
LnN1Y2Nlc3MgPT0gMTtcblx0JHBhZ2VzLmZvcm1hdCgnJGRhdGFcXG4nKTtcbkVO
RDtcbnEgPSBxdWVyeS5vcmlnIHwgdXJpO1xudG9vbHMucXVlcnkuYWRkKCdodHRw
czovL3d3dy5nb29nbGUuY29tL3NlYXJjaD9xPScgXyBxIF8gJyZzdGFydD0xMCZu
dW09MTAwJywgMSkgJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNG
aWxlTmFtZSI6IlslIGxpbmVzID0gbGluZXMgKyBwMS5zZXJwLnNpemU7XG5VU0Ug
TWF0aDtcbidzZXJwcy8nIF8gTWF0aC5pbnQobGluZXMgLyAxMDAwMDAwKSBfICcu
dHh0JyAlXSIsImFkZGl0aW9uYWxGb3JtYXRzIjpbWyJbJSBJRiBwMS5pbmZvLnN1
Y2Nlc3MgPT0gMTtcblx0VVNFIE1hdGg7XG5cdCdzZXJwX3Jhdy91c18nIF8gTWF0
aC5pbnQocXVlcnkubnVtIC8gMjAwMCkgXyAnLycgXyBxdWVyeS5maXJzdCBfICcu
aHRtbCc7XG5FTkQgJV0iLCIkcDEucGFnZXMuMC5kYXRhOyRwMS5wYWdlcy4xLmRh
dGEiXSxbInNlcnBfZmFpbC9mYWlsZWQudHh0IiwiWyUgSUYgcDEuaW5mby5zdWNj
ZXNzID09IDA7XG5cdHF1ZXJ5Lm9yaWcgXyBcIlxcblwiO1xuRU5EICVdIl1dLCJy
ZXN1bHRzVW5pcXVlIjoibm8iLCJxdWVyeUZvcm1hdCI6WyJodHRwczovL3d3dy5n
b29nbGUuY28udWsvc2VhcmNoP3E9WyUgcXVlcnkgfCB1cmkgJV0mcHdzPTAmZ2w9
dXMiXSwidW5pcXVlUXVlcmllcyI6dHJ1ZSwic2F2ZUZhaWxlZFF1ZXJpZXMiOmZh
bHNlLCJpdGVyYXRvck9wdGlvbnMiOnsib25BbGxMZXZlbHMiOmZhbHNlLCJxdWVy
eUJ1aWxkZXJzQWZ0ZXJJdGVyYXRvciI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNPbkFs
bExldmVscyI6ZmFsc2V9LCJyZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZh
bHNlfSwiZG9Mb2ciOiJubyIsImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25z
IjpmYWxzZSwicmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwi
cXVlcnlCdWlsZGVycyI6W10sInJlc3VsdHNCdWlsZGVycyI6W10sImNvbmZpZ092
ZXJyaWRlcyI6W10sInJ1blRhc2tPbkNvbXBsZXRlIjpudWxsLCJ1c2VSZXN1bHRz
RmlsZUFzUXVlcmllc0ZpbGUiOmZhbHNlLCJydW5UYXNrT25Db21wbGV0ZUNvbmZp
ZyI6ImRlZmF1bHQiLCJ0b29sc0pTIjoiIn19

 

[Support]

$pages should be used when the task has options Use pages and/or Check next page. In this case enough $data.
Also in the task is not a variable $serp.
To solve your problem, and on the basis of the above, the preset will look like this:

Spoiler: Preset

eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs
InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy
aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl
cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX
aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws
IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi
fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi
OiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHliYW5uZWRjbGVh
bnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InRpbWVv
dXQiLCJ2YWx1ZSI6IjMwIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslIElGIHAxLmlu
Zm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8gXCJcXG5cIjtcbkVORDtcbnEg
PSBxdWVyeS5vcmlnIHwgdXJpO1xudG9vbHMucXVlcnkuYWRkKCdodHRwczovL3d3
dy5nb29nbGUuY29tL3NlYXJjaD9xPScgXyBxIF8gJyZzdGFydD0xMCZudW09MTAw
JywgMSkgJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFt
ZSI6IlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRVU0UgTWF0aDtcblx0
J3NlcnBfcmF3L3VzXycgXyBNYXRoLmludChxdWVyeS5udW0gLyAyMDAwKSBfICcv
JyBfIHF1ZXJ5LmZpcnN0IF8gJy5odG1sJztcbkVORCAlXSIsImFkZGl0aW9uYWxG
b3JtYXRzIjpbWyJzZXJwX2ZhaWwvZmFpbGVkLnR4dCIsIlslIElGIHAxLmluZm8u
c3VjY2VzcyA9PSAwO1xuXHRxdWVyeS5vcmlnIF8gXCJcXG5cIjtcbkVORCAlXSJd
XSwicmVzdWx0c1VuaXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiaHR0cHM6Ly93
d3cuZ29vZ2xlLmNvLnVrL3NlYXJjaD9xPVslIHF1ZXJ5IHwgdXJpICVdJnB3cz0w
JmdsPXVzIl0sInVuaXF1ZVF1ZXJpZXMiOnRydWUsInNhdmVGYWlsZWRRdWVyaWVz
IjpmYWxzZSwiaXRlcmF0b3JPcHRpb25zIjp7Im9uQWxsTGV2ZWxzIjpmYWxzZSwi
cXVlcnlCdWlsZGVyc0FmdGVySXRlcmF0b3IiOmZhbHNlLCJxdWVyeUJ1aWxkZXJz
T25BbGxMZXZlbHMiOmZhbHNlfSwicmVzdWx0c09wdGlvbnMiOnsib3ZlcndyaXRl
IjpmYWxzZX0sImRvTG9nIjoibm8iLCJrZWVwVW5pcXVlIjoiTm8iLCJtb3JlT3B0
aW9ucyI6ZmFsc2UsInJlc3VsdHNQcmVwZW5kIjoiIiwicmVzdWx0c0FwcGVuZCI6
IiIsInF1ZXJ5QnVpbGRlcnMiOltdLCJyZXN1bHRzQnVpbGRlcnMiOltdLCJjb25m
aWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVz
dWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVD
b25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9fQ==

 

[scrapefun]

Thanks!

I made some updates where I could change the URL parameters on the fly by reading data from the query file and everything is working except for the file naming.

I'm pretty sure it has to do with the second query and how the variables are stored/not stored.

Spoiler: Task Screenshot

The data from both queries is written to the same file which I want but the variables I'm using in the result file name don't have any values.

If I change the result file name to the screenshot below file naming works for the first query only but it causes a completely different file to be written for the 2nd query.

How do you carry the parameters over to the second query so the variables are used in the file name?

Spoiler: Code Export

eyJwcmVzZXQiOiJjdXN0b20gZ29vZ2xlIiwidmFsdWUiOnsicHJlc2V0IjoiY3Vz
dG9tIGdvb2dsZSIsInBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsi
dHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0s
eyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1v
emlsbGEvNS4wIChXaW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUz
Ny4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNh
ZmFyaS81MzcuMzYifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRy
aWVzIiwidmFsdWUiOiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJv
eHliYW5uZWRjbGVhbnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUi
LCJpZCI6InRpbWVvdXQiLCJ2YWx1ZSI6IjMwIn0seyJ0eXBlIjoib3ZlcnJpZGUi
LCJpZCI6InVzZXByb3h5IiwidmFsdWUiOmZhbHNlfSx7InR5cGUiOiJvdmVycmlk
ZSIsImlkIjoicmVxdWVzdGRlbGF5IiwidmFsdWUiOiIwIn1dXSwicmVzdWx0c0Zv
cm1hdCI6IlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8g
XCJcXG5cIjtcbkVORDtcbnQgPSBxdWVyeS50bGQ7XG5nID0gcXVlcnkuZ2w7XG5o
ID0gcXVlcnkuaGw7XG5pID0gcXVlcnkuaWQ7XG5xID0gcXVlcnkua3cgfCB1cmk7
XG50b29scy5xdWVyeS5hZGQoJ2h0dHBzOi8vJyBfIHQgXyAnL3NlYXJjaD9xPScg
XyBxIF8gJyZnbD0nIF8gZyBfICcmaGw9JyBfIGggXyAnJnN0YXJ0PTEwJm51bT0x
MDAnLCAxKSAlXSIsInJlc3VsdHNTYXZlVG8iOiJmaWxlIiwicmVzdWx0c0ZpbGVO
YW1lIjoiWyUgSUYgcDEuaW5mby5zdWNjZXNzID09IDE7XG5cdFVTRSBNYXRoO1xu
XHQndGVzdF9yYXcvdXNfJyBfIE1hdGguaW50KHF1ZXJ5Lm51bSAvIDIwMDApIF8g
Jy8nIF8gcXVlcnkuZ2wgXyAnXycgXyBxdWVyeS5obCBfICdfJyBfIHF1ZXJ5Lmlk
IF8gJ18nIF8gcXVlcnkua3cgXyAnLmh0bWwnO1xuRU5EICVdIiwiYWRkaXRpb25h
bEZvcm1hdHMiOltbInRlc3RfZmFpbC9mYWlsZWQudHh0IiwiWyUgSUYgcDEuaW5m
by5zdWNjZXNzID09IDA7XG5cdHF1ZXJ5Lm9yaWcgXyBcIlxcblwiO1xuRU5EICVd
Il0sWyIkZGF0ZWZpbGUuZm9ybWF0KCkudHh0IiwiJHAxLnByZXNldCJdXSwicmVz
dWx0c1VuaXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiaHR0cHM6Ly9bJSBxdWVy
eS50bGQgJV0vc2VhcmNoP3E9WyUgcXVlcnkua3cgfCB1cmkgJV0mcHdzPTAmZ2w9
WyUgcXVlcnkuZ2wgJV0maGw9WyUgcXVlcnkuaGwgJV0iXSwidW5pcXVlUXVlcmll
cyI6dHJ1ZSwic2F2ZUZhaWxlZFF1ZXJpZXMiOmZhbHNlLCJpdGVyYXRvck9wdGlv
bnMiOnsib25BbGxMZXZlbHMiOmZhbHNlLCJxdWVyeUJ1aWxkZXJzQWZ0ZXJJdGVy
YXRvciI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNPbkFsbExldmVscyI6ZmFsc2V9LCJy
ZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZhbHNlfSwiZG9Mb2ciOiJubyIs
ImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25zIjpmYWxzZSwicmVzdWx0c1By
ZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwicXVlcnlCdWlsZGVycyI6W3si
c291cmNlIjoicXVlcnkiLCJ0eXBlIjoic3RyaW5nU3BsaXQiLCJzZXBhcmF0b3Ii
OiI7IiwidG8iOlsidGxkIiwiZ2wiLCJobCIsImlkIiwia3ciXX1dLCJyZXN1bHRz
QnVpbGRlcnMiOltdLCJjb25maWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21w
bGV0ZSI6bnVsbCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwi
cnVuVGFza09uQ29tcGxldGVDb25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9
fQ==

 

[Support]

On the 1st level (2nd Page) need to submit the same query as in the 0th, in order to could work Query Builder. Besides the option Query Builders on all levels must be enabled. And link needs to be formed in the General query format, depending on the query level ($query.lvl). Also, the option Use format on all levels must be enabled. Therefore, the preset will look like this:

Spoiler: Preset

eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs
InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy
aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl
cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX
aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws
IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi
fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi
OiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHliYW5uZWRjbGVh
bnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InRpbWVv
dXQiLCJ2YWx1ZSI6IjMwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXBy
b3h5IiwidmFsdWUiOmZhbHNlfSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicmVx
dWVzdGRlbGF5IiwidmFsdWUiOiIwIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslIElG
IHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8gXCJcXG5cIjtcbkVO
RDtcbnRvb2xzLnF1ZXJ5LmFkZChxdWVyeS5maXJzdCwgMSkgJV0iLCJyZXN1bHRz
U2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6IlslIElGIHAxLmluZm8u
c3VjY2VzcyA9PSAxO1xuXHRVU0UgTWF0aDtcblx0J3Rlc3RfcmF3L3VzXycgXyBN
YXRoLmludChxdWVyeS5udW0gLyAyMDAwKSBfICcvJyBfIHF1ZXJ5LmdsIF8gJ18n
IF8gcXVlcnkuaGwgXyAnXycgXyBxdWVyeS5pZCBfICdfJyBfIHF1ZXJ5Lmt3IF8g
Jy5odG1sJztcbkVORCAlXSIsImFkZGl0aW9uYWxGb3JtYXRzIjpbWyJ0ZXN0X2Zh
aWwvZmFpbGVkLnR4dCIsIlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAwO1xuXHRx
dWVyeS5vcmlnIF8gXCJcXG5cIjtcbkVORCAlXSJdLFsiJGRhdGVmaWxlLmZvcm1h
dCgpLnR4dCIsIiRwMS5wcmVzZXQiXV0sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1
ZXJ5Rm9ybWF0IjpbIlslIHEgPSBxdWVyeS5rdyB8IHVyaTtcbklGIHF1ZXJ5Lmx2
bCA9PSAwO1xuXHQnaHR0cHM6Ly8nIF8gcXVlcnkudGxkIF8gJy9zZWFyY2g/cT0n
IF8gcSBfICcmcHdzPTAmZ2w9JyBfIHF1ZXJ5LmdsIF8gJyZobD0nIF8gcXVlcnku
aGw7XG5FTFNJRiBxdWVyeS5sdmwgPT0gMTtcblx0J2h0dHBzOi8vJyBfIHF1ZXJ5
LnRsZCBfICcvc2VhcmNoP3E9JyBfIHEgXyAnJmdsPScgXyBxdWVyeS5nbCBfICcm
aGw9JyBfIHF1ZXJ5LmhsIF8gJyZzdGFydD0xMCZudW09MTAwJztcbkVORCAlXSJd
LCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2Us
Iml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExldmVscyI6dHJ1ZSwicXVlcnlCdWls
ZGVyc0FmdGVySXRlcmF0b3IiOmZhbHNlLCJxdWVyeUJ1aWxkZXJzT25BbGxMZXZl
bHMiOnRydWV9LCJyZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZhbHNlfSwi
ZG9Mb2ciOiJubyIsImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25zIjpmYWxz
ZSwicmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwicXVlcnlC
dWlsZGVycyI6W3sic291cmNlIjoicXVlcnkiLCJ0eXBlIjoic3RyaW5nU3BsaXQi
LCJzZXBhcmF0b3IiOiI7IiwidG8iOlsidGxkIiwiZ2wiLCJobCIsImlkIiwia3ci
XX1dLCJyZXN1bHRzQnVpbGRlcnMiOltdLCJjb25maWdPdmVycmlkZXMiOltdLCJy
dW5UYXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNG
aWxlIjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVDb25maWciOiJkZWZhdWx0Iiwi
dG9vbHNKUyI6IiJ9fQ==

 

[scrapefun]

Thanks! Never would have figured that out. Saved the day again. Thanks for all of your help and patience!

 

[scrapefun]

Ugh, another quirky issue cropped up.

I have it set to so that every 2000 queries a new directory is created to keep the files per directory under control. The problem is sometimes the two queries per keyword are saved in two different files in different directories. Is there any way to ensure just a single file per query is created?

If that is not possible how would you set it up to save each query to a separate file?

 

[Support]

This occurs because the 1st page can be processed, for example, under the number 1990, and the second - under 2057. Therefore, in the current configuration file may be located in different folders.
There is a solution. It is necessary to generate name of the folder on the first page and send it to the second, there to be used. So here are working preset:

Spoiler: Code

eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs
InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy
aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl
cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX
aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws
IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi
fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi
OiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHliYW5uZWRjbGVh
bnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InRpbWVv
dXQiLCJ2YWx1ZSI6IjMwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXBy
b3h5IiwidmFsdWUiOmZhbHNlfSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicmVx
dWVzdGRlbGF5IiwidmFsdWUiOiIwIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslIElG
IHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8gXCJcXG5cIjtcbkVO
RDtcblVTRSBNYXRoO1xuZGlyID0gTWF0aC5pbnQocXVlcnkubnVtIC8gMjAwMCk7
XG50b29scy5xdWVyeS5hZGQocXVlcnkuZmlyc3QgXyAnOycgXyBkaXIsIDEpICVd
IiwicmVzdWx0c1NhdmVUbyI6ImZpbGUiLCJyZXN1bHRzRmlsZU5hbWUiOiJbJSBJ
RiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMTtcblx0SUYgcXVlcnkubHZsID09IDA7XG4g
ICAgXHRVU0UgTWF0aDtcbiAgICAgICAgZGlyID0gTWF0aC5pbnQocXVlcnkubnVt
IC8gMjAwMCk7XG4gICAgRUxTSUYgcXVlcnkubHZsID09IDE7XG4gICAgXHRkaXIg
PSBxdWVyeS5kaXI7XG4gICAgRU5EO1xuXHQndGVzdF9yYXcvdXNfJyBfIGRpciBf
ICcvJyBfIHF1ZXJ5LmdsIF8gJ18nIF8gcXVlcnkuaGwgXyAnXycgXyBxdWVyeS5p
ZCBfICdfJyBfIHF1ZXJ5Lmt3IF8gJy5odG1sJztcbkVORCAlXSIsImFkZGl0aW9u
YWxGb3JtYXRzIjpbWyJ0ZXN0X2ZhaWwvZmFpbGVkLnR4dCIsIlslIElGIHAxLmlu
Zm8uc3VjY2VzcyA9PSAwO1xuXHRxdWVyeS5vcmlnIF8gXCJcXG5cIjtcbkVORCAl
XSJdLFsiJGRhdGVmaWxlLmZvcm1hdCgpLnR4dCIsIiRwMS5wcmVzZXQiXV0sInJl
c3VsdHNVbmlxdWUiOiJubyIsInF1ZXJ5Rm9ybWF0IjpbIlslIHEgPSBxdWVyeS5r
dyB8IHVyaTtcbklGIHF1ZXJ5Lmx2bCA9PSAwO1xuXHQnaHR0cHM6Ly8nIF8gcXVl
cnkudGxkIF8gJy9zZWFyY2g/cT0nIF8gcSBfICcmcHdzPTAmZ2w9JyBfIHF1ZXJ5
LmdsIF8gJyZobD0nIF8gcXVlcnkuaGw7XG5FTFNJRiBxdWVyeS5sdmwgPT0gMTtc
blx0J2h0dHBzOi8vJyBfIHF1ZXJ5LnRsZCBfICcvc2VhcmNoP3E9JyBfIHEgXyAn
JmdsPScgXyBxdWVyeS5nbCBfICcmaGw9JyBfIHF1ZXJ5LmhsIF8gJyZzdGFydD0x
MCZudW09MTAwJztcbkVORCAlXSJdLCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZl
RmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExl
dmVscyI6dHJ1ZSwicXVlcnlCdWlsZGVyc0FmdGVySXRlcmF0b3IiOmZhbHNlLCJx
dWVyeUJ1aWxkZXJzT25BbGxMZXZlbHMiOnRydWV9LCJyZXN1bHRzT3B0aW9ucyI6
eyJvdmVyd3JpdGUiOmZhbHNlfSwiZG9Mb2ciOiJubyIsImtlZXBVbmlxdWUiOiJO
byIsIm1vcmVPcHRpb25zIjpmYWxzZSwicmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1
bHRzQXBwZW5kIjoiIiwicXVlcnlCdWlsZGVycyI6W3sic291cmNlIjoicXVlcnki
LCJ0eXBlIjoic3RyaW5nU3BsaXQiLCJzZXBhcmF0b3IiOiI7IiwidG8iOlsidGxk
IiwiZ2wiLCJobCIsImlkIiwia3ciLCJkaXIiXX1dLCJyZXN1bHRzQnVpbGRlcnMi
OltdLCJjb25maWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21wbGV0ZSI6bnVs
bCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwicnVuVGFza09u
Q29tcGxldGVDb25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9fQ==