When using Net:HTTP parser to scrape Google is it possible to change the URL parameters used for the second page? So the first query would be normal first page Google query: https://www.google.com/search?q=$query But for the second/next page I want the query string to be: https://www.google.com/search?q=$query&start=10&num=100 Thanks!
Use $tools.query.add: http://en.a-parser.com/wiki/template-tools/#tools-query-adding-of-requests to add your own query. In this case, it would look like this: Code: [% tools.query.add('https://www.google.com/search?q=' _ query _ '&start=10&num=100') %]
Where would I add that to this Task: Spoiler: Task Screenshot Spoiler: Task Import Code eyJwcmVzZXQiOiJUd28gUGFnZSBUZXN0IiwidmFsdWUiOnsicHJlc2V0IjoiVHdv IFBhZ2UgVGVzdCIsInBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsi dHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0s eyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1v emlsbGEvNS4wIChXaW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUz Ny4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNh ZmFyaS81MzcuMzYifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicXVlcnlmb3Jt YXQiLCJ2YWx1ZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZS5jby51ay9zZWFyY2g/cT1b JSBxdWVyeSB8IHVyaSAlXSZwd3M9MCZnbD11cyJ9LHsidHlwZSI6Im92ZXJyaWRl IiwiaWQiOiJwcm94eXJldHJpZXMiLCJ2YWx1ZSI6IjEwMCJ9LHsidHlwZSI6ImN1 c3RvbVJlc3VsdCIsInJlc3VsdCI6WyJwYWdlcyIsImRhdGEiXSwicmVnZXgiOiIo Pzo8ZGl2IGNsYXNzPVwic3JnXCI+fDxkaXYgY2xhc3M9XCJnXCIgZGF0YS1odmVp ZD1cIlxcZCtcIj4pKC4rPyk8IS0tbi0tPjxcXC9kaXY+PFxcL2Rpdj4iLCJyZWdl eFR5cGUiOiJnIiwicmVzdWx0VHlwZSI6ImFycmF5IiwiYXJyYXlOYW1lIjoic3Jn IiwicmVzdWx0cyI6WyJkYXRhIl19LHsidHlwZSI6ImN1c3RvbVJlc3VsdCIsInJl c3VsdCI6WyJzcmciLCJkYXRhIl0sInJlZ2V4IjoiPGgzIGNsYXNzPVwiclwiPjxh IGhyZWY9XCIoLis/KVwiLis/JyhcXGQrKScuKz8+KC4rPyk8XFwvYT4iLCJyZWdl eFR5cGUiOiJnIiwicmVzdWx0VHlwZSI6ImFycmF5IiwiYXJyYXlOYW1lIjoic2Vy cCIsInJlc3VsdHMiOlsibGluayIsInJhbmsiLCJ0aXRsZSJdfSx7InR5cGUiOiJv cHRpb25zIiwiaWQiOiJjaGVja05leHRQYWdlIiwidmFsdWUiOiI8dGQgY2xhc3M9 XCJiIG5hdmVuZFwiPjxhIGNsYXNzPVwicG5cIiBocmVmPVwiKFteXCJdKylcIiBp ZD1cInBubmV4dFwiIiwiYWRkaXRpb25hbCI6eyJjaGVja05leHRQYWdlTGltaXQi OiIxIn19LHsidHlwZSI6ImN1c3RvbVJlc3VsdCIsInJlc3VsdCI6WyJwYWdlcyIs ImRhdGEiXSwicmVnZXgiOiI8c3BhbiBjbGFzcz1cInNwZWxsXCI+U2hvd2luZyBy ZXN1bHRzIGZvcjxcXC9zcGFuPiA8YS4rPz4oLis/KTxcXC9hPiIsInJlZ2V4VHlw ZSI6ImkiLCJyZXN1bHRUeXBlIjoiYXJyYXkiLCJhcnJheU5hbWUiOiJtaXNzcGVs bCIsInJlc3VsdHMiOlsia2V5Il19LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJw cm94eWJhbm5lZGNsZWFudXAiLCJ2YWx1ZSI6IjAifSx7InR5cGUiOiJvdmVycmlk ZSIsImlkIjoidGltZW91dCIsInZhbHVlIjoiMzAifV1dLCJyZXN1bHRzRm9ybWF0 IjoiWyUgSUYgcDEuaW5mby5zdWNjZXNzID09IDE7XG5cdEZPUkVBQ0ggcDEuc2Vy cDtcblx0XHRtcCA9IChwMS5taXNzcGVsbC4wLmtleSA9PSAnJykgPyAnMCcgOiAn MSc7XG5cdFx0cXVlcnkub3JpZyBfICc7ICcgXyBtcCBfICc7ICcgXyBsb29wLmNv dW50IF8gJzsgJyBfIGxpbmsgXyBcIlxcblwiO1xuXHRFTkQ7XG5FTkQgJV0iLCJy ZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6IlslIGxpbmVz ID0gbGluZXMgKyBwMS5zZXJwLnNpemU7XG5VU0UgTWF0aDtcbidzZXJwcy8nIF8g TWF0aC5pbnQobGluZXMgLyAxMDAwMDAwKSBfICcudHh0JyAlXSIsImFkZGl0aW9u YWxGb3JtYXRzIjpbWyJbJSBJRiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMTtcblx0VVNF IE1hdGg7XG5cdCdzZXJwX3Jhdy8xXycgXyBNYXRoLmludChxdWVyeS5udW0gLyAy MDAwKSBfICcvJyBfIHF1ZXJ5Lm9yaWcgXyAnJztcbkVORCAlXSIsIiRwMS5wYWdl cy4wLmRhdGEiXSxbIlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRVU0Ug TWF0aDtcblx0J3NlcnBfcmF3LzFfJyBfIE1hdGguaW50KHF1ZXJ5Lm51bSAvIDIw MDApIF8gJy8nIF8gcXVlcnkub3JpZyBfICcnO1xuRU5EICVdIiwiJHAxLnBhZ2Vz LjEuZGF0YSJdLFsic2VycF9mYWlsL2ZhaWxlZC50eHQiLCJbJSBJRiBwMS5pbmZv LnN1Y2Nlc3MgPT0gMDtcblx0cXVlcnkub3JpZyBfIFwiXFxuXCI7XG5FTkQgJV0i XV0sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1ZXJ5Rm9ybWF0IjpbIiRxdWVyeSJd LCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2Us Iml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExldmVscyI6ZmFsc2UsInF1ZXJ5QnVp bGRlcnNBZnRlckl0ZXJhdG9yIjpmYWxzZSwicXVlcnlCdWlsZGVyc09uQWxsTGV2 ZWxzIjpmYWxzZX0sInJlc3VsdHNPcHRpb25zIjp7Im92ZXJ3cml0ZSI6ZmFsc2V9 LCJkb0xvZyI6Im5vIiwia2VlcFVuaXF1ZSI6Ik5vIiwibW9yZU9wdGlvbnMiOmZh bHNlLCJyZXN1bHRzUHJlcGVuZCI6IiIsInJlc3VsdHNBcHBlbmQiOiIiLCJxdWVy eUJ1aWxkZXJzIjpbXSwicmVzdWx0c0J1aWxkZXJzIjpbeyJzb3VyY2UiOlswLFsi bWlzc3BlbGwiLCJrZXkiXV0sInR5cGUiOiJyZW1vdmVIdG1sIiwiYXJyYXkiOiJt aXNzcGVsbCIsInRvIjoia2V5In1dLCJjb25maWdPdmVycmlkZXMiOltdLCJydW5U YXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNGaWxl IjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVDb25maWciOiJkZWZhdWx0IiwidG9v bHNKUyI6IiJ9fQ==
I don't understand how to integrate it into my existing result format and do I still need to use the "Next Page" regex that I'm using or can I remove that since the $tools.query.add code seems to take care of the second query I need.
In this case, the result format should look like this: At the same time query template need to move from a Query format of preset into the General query format to avoid formatting the request obtained from $tools.query.add. Also need to remove Check next page, because this feature is already implementing $tools.query.add. Spoiler: Code Code: eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi OiIxMDAifSx7InR5cGUiOiJjdXN0b21SZXN1bHQiLCJyZXN1bHQiOlsicGFnZXMi LCJkYXRhIl0sInJlZ2V4IjoiKD86PGRpdiBjbGFzcz1cInNyZ1wiPnw8ZGl2IGNs YXNzPVwiZ1wiIGRhdGEtaHZlaWQ9XCJcXGQrXCI+KSguKz8pPCEtLW4tLT48XFwv ZGl2PjxcXC9kaXY+IiwicmVnZXhUeXBlIjoiZyIsInJlc3VsdFR5cGUiOiJhcnJh eSIsImFycmF5TmFtZSI6InNyZyIsInJlc3VsdHMiOlsiZGF0YSJdfSx7InR5cGUi OiJjdXN0b21SZXN1bHQiLCJyZXN1bHQiOlsic3JnIiwiZGF0YSJdLCJyZWdleCI6 IjxoMyBjbGFzcz1cInJcIj48YSBocmVmPVwiKC4rPylcIi4rPycoXFxkKyknLis/ PiguKz8pPFxcL2E+IiwicmVnZXhUeXBlIjoiZyIsInJlc3VsdFR5cGUiOiJhcnJh eSIsImFycmF5TmFtZSI6InNlcnAiLCJyZXN1bHRzIjpbImxpbmsiLCJyYW5rIiwi dGl0bGUiXX0seyJ0eXBlIjoiY3VzdG9tUmVzdWx0IiwicmVzdWx0IjpbInBhZ2Vz IiwiZGF0YSJdLCJyZWdleCI6IjxzcGFuIGNsYXNzPVwic3BlbGxcIj5TaG93aW5n IHJlc3VsdHMgZm9yPFxcL3NwYW4+IDxhLis/PiguKz8pPFxcL2E+IiwicmVnZXhU eXBlIjoiaSIsInJlc3VsdFR5cGUiOiJhcnJheSIsImFycmF5TmFtZSI6Im1pc3Nw ZWxsIiwicmVzdWx0cyI6WyJrZXkiXX0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6 InByb3h5YmFubmVkY2xlYW51cCIsInZhbHVlIjoiMCJ9LHsidHlwZSI6Im92ZXJy aWRlIiwiaWQiOiJ0aW1lb3V0IiwidmFsdWUiOiIzMCJ9XV0sInJlc3VsdHNGb3Jt YXQiOiJbJSBJRiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMTtcblx0Rk9SRUFDSCBwMS5z ZXJwO1xuXHRcdG1wID0gKHAxLm1pc3NwZWxsLjAua2V5ID09ICcnKSA/ICcwJyA6 ICcxJztcblx0XHRxdWVyeS5vcmlnIF8gJzsgJyBfIG1wIF8gJzsgJyBfIGxvb3Au Y291bnQgXyAnOyAnIF8gbGluayBfIFwiXFxuXCI7XG5cdEVORDtcbkVORDtcbnEg PSBxdWVyeS5vcmlnIHwgdXJpO1xudG9vbHMucXVlcnkuYWRkKCdodHRwczovL3d3 dy5nb29nbGUuY29tL3NlYXJjaD9xPScgXyBxIF8gJyZzdGFydD0xMCZudW09MTAw JywgMSkgJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFt ZSI6IlslIGxpbmVzID0gbGluZXMgKyBwMS5zZXJwLnNpemU7XG5VU0UgTWF0aDtc bidzZXJwcy8nIF8gTWF0aC5pbnQobGluZXMgLyAxMDAwMDAwKSBfICcudHh0JyAl XSIsImFkZGl0aW9uYWxGb3JtYXRzIjpbWyJbJSBJRiBwMS5pbmZvLnN1Y2Nlc3Mg PT0gMTtcblx0VVNFIE1hdGg7XG5cdCdzZXJwX3Jhdy8xXycgXyBNYXRoLmludChx dWVyeS5udW0gLyAyMDAwKSBfICcvJyBfIHF1ZXJ5Lm9yaWcgXyAnJztcbkVORCAl XSIsIiRwMS5wYWdlcy4wLmRhdGEiXSxbIlslIElGIHAxLmluZm8uc3VjY2VzcyA9 PSAxO1xuXHRVU0UgTWF0aDtcblx0J3NlcnBfcmF3LzFfJyBfIE1hdGguaW50KHF1 ZXJ5Lm51bSAvIDIwMDApIF8gJy8nIF8gcXVlcnkub3JpZyBfICcnO1xuRU5EICVd IiwiJHAxLnBhZ2VzLjEuZGF0YSJdLFsic2VycF9mYWlsL2ZhaWxlZC50eHQiLCJb JSBJRiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMDtcblx0cXVlcnkub3JpZyBfIFwiXFxu XCI7XG5FTkQgJV0iXV0sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1ZXJ5Rm9ybWF0 IjpbImh0dHBzOi8vd3d3Lmdvb2dsZS5jby51ay9zZWFyY2g/cT1bJSBxdWVyeSB8 IHVyaSAlXSZwd3M9MCZnbD11cyJdLCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZl RmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExl dmVscyI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNBZnRlckl0ZXJhdG9yIjpmYWxzZSwi cXVlcnlCdWlsZGVyc09uQWxsTGV2ZWxzIjpmYWxzZX0sInJlc3VsdHNPcHRpb25z Ijp7Im92ZXJ3cml0ZSI6ZmFsc2V9LCJkb0xvZyI6Im5vIiwia2VlcFVuaXF1ZSI6 Ik5vIiwibW9yZU9wdGlvbnMiOmZhbHNlLCJyZXN1bHRzUHJlcGVuZCI6IiIsInJl c3VsdHNBcHBlbmQiOiIiLCJxdWVyeUJ1aWxkZXJzIjpbXSwicmVzdWx0c0J1aWxk ZXJzIjpbeyJzb3VyY2UiOlswLFsibWlzc3BlbGwiLCJrZXkiXV0sInR5cGUiOiJy ZW1vdmVIdG1sIiwiYXJyYXkiOiJtaXNzcGVsbCIsInRvIjoia2V5In1dLCJjb25m aWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVz dWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVD b25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9fQ==
Thanks! This works great. I'm trying to setup a preset like the one I posted above except it only saves the raw html results and not the raw and parsed results. I have it working but I can't figure out how to set it up without the 1st result file for the parsed results. I highlighted the result file I want to remove in the screenshot below. It does work as is but it would be great to completely remove the unnecessary result file entry. When I remove that result file it doesn't work. I am guessing the problem is with the result format. Spoiler: Preset Code eyJwcmVzZXQiOiJyb3VnaDMiLCJ2YWx1ZSI6eyJwcmVzZXQiOiJyb3VnaDMiLCJw YXJzZXJzIjpbWyJOZXQ6OkhUVFAiLCJkZWZhdWx0Iix7InR5cGUiOiJvdmVycmlk ZSIsImlkIjoiZ29vZENvZGUiLCJ2YWx1ZSI6WzIwMF19LHsidHlwZSI6Im92ZXJy aWRlIiwiaWQiOiJ1c2VyLWFnZW50IiwidmFsdWUiOiJNb3ppbGxhLzUuMCAoV2lu ZG93cyBOVCA2LjE7IFdPVzY0KSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBs aWtlIEdlY2tvKSBDaHJvbWUvNDcuMC4yNTI2LjExMSBTYWZhcmkvNTM3LjM2In0s eyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InByb3h5cmV0cmllcyIsInZhbHVlIjoi MTAwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InByb3h5YmFubmVkY2xlYW51 cCIsInZhbHVlIjoiMCJ9LHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJ0aW1lb3V0 IiwidmFsdWUiOiIzMCJ9XV0sInJlc3VsdHNGb3JtYXQiOiJbJSBJRiBwMS5pbmZv LnN1Y2Nlc3MgPT0gMTtcblx0JHBhZ2VzLmZvcm1hdCgnJGRhdGFcXG4nKTtcbkVO RDtcbnEgPSBxdWVyeS5vcmlnIHwgdXJpO1xudG9vbHMucXVlcnkuYWRkKCdodHRw czovL3d3dy5nb29nbGUuY29tL3NlYXJjaD9xPScgXyBxIF8gJyZzdGFydD0xMCZu dW09MTAwJywgMSkgJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNG aWxlTmFtZSI6IlslIGxpbmVzID0gbGluZXMgKyBwMS5zZXJwLnNpemU7XG5VU0Ug TWF0aDtcbidzZXJwcy8nIF8gTWF0aC5pbnQobGluZXMgLyAxMDAwMDAwKSBfICcu dHh0JyAlXSIsImFkZGl0aW9uYWxGb3JtYXRzIjpbWyJbJSBJRiBwMS5pbmZvLnN1 Y2Nlc3MgPT0gMTtcblx0VVNFIE1hdGg7XG5cdCdzZXJwX3Jhdy91c18nIF8gTWF0 aC5pbnQocXVlcnkubnVtIC8gMjAwMCkgXyAnLycgXyBxdWVyeS5maXJzdCBfICcu aHRtbCc7XG5FTkQgJV0iLCIkcDEucGFnZXMuMC5kYXRhOyRwMS5wYWdlcy4xLmRh dGEiXSxbInNlcnBfZmFpbC9mYWlsZWQudHh0IiwiWyUgSUYgcDEuaW5mby5zdWNj ZXNzID09IDA7XG5cdHF1ZXJ5Lm9yaWcgXyBcIlxcblwiO1xuRU5EICVdIl1dLCJy ZXN1bHRzVW5pcXVlIjoibm8iLCJxdWVyeUZvcm1hdCI6WyJodHRwczovL3d3dy5n b29nbGUuY28udWsvc2VhcmNoP3E9WyUgcXVlcnkgfCB1cmkgJV0mcHdzPTAmZ2w9 dXMiXSwidW5pcXVlUXVlcmllcyI6dHJ1ZSwic2F2ZUZhaWxlZFF1ZXJpZXMiOmZh bHNlLCJpdGVyYXRvck9wdGlvbnMiOnsib25BbGxMZXZlbHMiOmZhbHNlLCJxdWVy eUJ1aWxkZXJzQWZ0ZXJJdGVyYXRvciI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNPbkFs bExldmVscyI6ZmFsc2V9LCJyZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZh bHNlfSwiZG9Mb2ciOiJubyIsImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25z IjpmYWxzZSwicmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwi cXVlcnlCdWlsZGVycyI6W10sInJlc3VsdHNCdWlsZGVycyI6W10sImNvbmZpZ092 ZXJyaWRlcyI6W10sInJ1blRhc2tPbkNvbXBsZXRlIjpudWxsLCJ1c2VSZXN1bHRz RmlsZUFzUXVlcmllc0ZpbGUiOmZhbHNlLCJydW5UYXNrT25Db21wbGV0ZUNvbmZp ZyI6ImRlZmF1bHQiLCJ0b29sc0pTIjoiIn19
$pages should be used when the task has options Use pages and/or Check next page. In this case enough $data. Also in the task is not a variable $serp. To solve your problem, and on the basis of the above, the preset will look like this: Spoiler: Preset Code: eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi OiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHliYW5uZWRjbGVh bnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InRpbWVv dXQiLCJ2YWx1ZSI6IjMwIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslIElGIHAxLmlu Zm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8gXCJcXG5cIjtcbkVORDtcbnEg PSBxdWVyeS5vcmlnIHwgdXJpO1xudG9vbHMucXVlcnkuYWRkKCdodHRwczovL3d3 dy5nb29nbGUuY29tL3NlYXJjaD9xPScgXyBxIF8gJyZzdGFydD0xMCZudW09MTAw JywgMSkgJV0iLCJyZXN1bHRzU2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFt ZSI6IlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRVU0UgTWF0aDtcblx0 J3NlcnBfcmF3L3VzXycgXyBNYXRoLmludChxdWVyeS5udW0gLyAyMDAwKSBfICcv JyBfIHF1ZXJ5LmZpcnN0IF8gJy5odG1sJztcbkVORCAlXSIsImFkZGl0aW9uYWxG b3JtYXRzIjpbWyJzZXJwX2ZhaWwvZmFpbGVkLnR4dCIsIlslIElGIHAxLmluZm8u c3VjY2VzcyA9PSAwO1xuXHRxdWVyeS5vcmlnIF8gXCJcXG5cIjtcbkVORCAlXSJd XSwicmVzdWx0c1VuaXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiaHR0cHM6Ly93 d3cuZ29vZ2xlLmNvLnVrL3NlYXJjaD9xPVslIHF1ZXJ5IHwgdXJpICVdJnB3cz0w JmdsPXVzIl0sInVuaXF1ZVF1ZXJpZXMiOnRydWUsInNhdmVGYWlsZWRRdWVyaWVz IjpmYWxzZSwiaXRlcmF0b3JPcHRpb25zIjp7Im9uQWxsTGV2ZWxzIjpmYWxzZSwi cXVlcnlCdWlsZGVyc0FmdGVySXRlcmF0b3IiOmZhbHNlLCJxdWVyeUJ1aWxkZXJz T25BbGxMZXZlbHMiOmZhbHNlfSwicmVzdWx0c09wdGlvbnMiOnsib3ZlcndyaXRl IjpmYWxzZX0sImRvTG9nIjoibm8iLCJrZWVwVW5pcXVlIjoiTm8iLCJtb3JlT3B0 aW9ucyI6ZmFsc2UsInJlc3VsdHNQcmVwZW5kIjoiIiwicmVzdWx0c0FwcGVuZCI6 IiIsInF1ZXJ5QnVpbGRlcnMiOltdLCJyZXN1bHRzQnVpbGRlcnMiOltdLCJjb25m aWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVz dWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVD b25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9fQ==
Thanks! I made some updates where I could change the URL parameters on the fly by reading data from the query file and everything is working except for the file naming. I'm pretty sure it has to do with the second query and how the variables are stored/not stored. Spoiler: Task Screenshot The data from both queries is written to the same file which I want but the variables I'm using in the result file name don't have any values. If I change the result file name to the screenshot below file naming works for the first query only but it causes a completely different file to be written for the 2nd query. How do you carry the parameters over to the second query so the variables are used in the file name? Spoiler: Code Export eyJwcmVzZXQiOiJjdXN0b20gZ29vZ2xlIiwidmFsdWUiOnsicHJlc2V0IjoiY3Vz dG9tIGdvb2dsZSIsInBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsi dHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0s eyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1v emlsbGEvNS4wIChXaW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUz Ny4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNh ZmFyaS81MzcuMzYifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRy aWVzIiwidmFsdWUiOiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJv eHliYW5uZWRjbGVhbnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUi LCJpZCI6InRpbWVvdXQiLCJ2YWx1ZSI6IjMwIn0seyJ0eXBlIjoib3ZlcnJpZGUi LCJpZCI6InVzZXByb3h5IiwidmFsdWUiOmZhbHNlfSx7InR5cGUiOiJvdmVycmlk ZSIsImlkIjoicmVxdWVzdGRlbGF5IiwidmFsdWUiOiIwIn1dXSwicmVzdWx0c0Zv cm1hdCI6IlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8g XCJcXG5cIjtcbkVORDtcbnQgPSBxdWVyeS50bGQ7XG5nID0gcXVlcnkuZ2w7XG5o ID0gcXVlcnkuaGw7XG5pID0gcXVlcnkuaWQ7XG5xID0gcXVlcnkua3cgfCB1cmk7 XG50b29scy5xdWVyeS5hZGQoJ2h0dHBzOi8vJyBfIHQgXyAnL3NlYXJjaD9xPScg XyBxIF8gJyZnbD0nIF8gZyBfICcmaGw9JyBfIGggXyAnJnN0YXJ0PTEwJm51bT0x MDAnLCAxKSAlXSIsInJlc3VsdHNTYXZlVG8iOiJmaWxlIiwicmVzdWx0c0ZpbGVO YW1lIjoiWyUgSUYgcDEuaW5mby5zdWNjZXNzID09IDE7XG5cdFVTRSBNYXRoO1xu XHQndGVzdF9yYXcvdXNfJyBfIE1hdGguaW50KHF1ZXJ5Lm51bSAvIDIwMDApIF8g Jy8nIF8gcXVlcnkuZ2wgXyAnXycgXyBxdWVyeS5obCBfICdfJyBfIHF1ZXJ5Lmlk IF8gJ18nIF8gcXVlcnkua3cgXyAnLmh0bWwnO1xuRU5EICVdIiwiYWRkaXRpb25h bEZvcm1hdHMiOltbInRlc3RfZmFpbC9mYWlsZWQudHh0IiwiWyUgSUYgcDEuaW5m by5zdWNjZXNzID09IDA7XG5cdHF1ZXJ5Lm9yaWcgXyBcIlxcblwiO1xuRU5EICVd Il0sWyIkZGF0ZWZpbGUuZm9ybWF0KCkudHh0IiwiJHAxLnByZXNldCJdXSwicmVz dWx0c1VuaXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiaHR0cHM6Ly9bJSBxdWVy eS50bGQgJV0vc2VhcmNoP3E9WyUgcXVlcnkua3cgfCB1cmkgJV0mcHdzPTAmZ2w9 WyUgcXVlcnkuZ2wgJV0maGw9WyUgcXVlcnkuaGwgJV0iXSwidW5pcXVlUXVlcmll cyI6dHJ1ZSwic2F2ZUZhaWxlZFF1ZXJpZXMiOmZhbHNlLCJpdGVyYXRvck9wdGlv bnMiOnsib25BbGxMZXZlbHMiOmZhbHNlLCJxdWVyeUJ1aWxkZXJzQWZ0ZXJJdGVy YXRvciI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNPbkFsbExldmVscyI6ZmFsc2V9LCJy ZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZhbHNlfSwiZG9Mb2ciOiJubyIs ImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25zIjpmYWxzZSwicmVzdWx0c1By ZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwicXVlcnlCdWlsZGVycyI6W3si c291cmNlIjoicXVlcnkiLCJ0eXBlIjoic3RyaW5nU3BsaXQiLCJzZXBhcmF0b3Ii OiI7IiwidG8iOlsidGxkIiwiZ2wiLCJobCIsImlkIiwia3ciXX1dLCJyZXN1bHRz QnVpbGRlcnMiOltdLCJjb25maWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21w bGV0ZSI6bnVsbCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwi cnVuVGFza09uQ29tcGxldGVDb25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9 fQ==
On the 1st level (2nd Page) need to submit the same query as in the 0th, in order to could work Query Builder. Besides the option Query Builders on all levels must be enabled. And link needs to be formed in the General query format, depending on the query level ($query.lvl). Also, the option Use format on all levels must be enabled. Therefore, the preset will look like this: Spoiler: Preset Code: eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi OiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHliYW5uZWRjbGVh bnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InRpbWVv dXQiLCJ2YWx1ZSI6IjMwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXBy b3h5IiwidmFsdWUiOmZhbHNlfSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicmVx dWVzdGRlbGF5IiwidmFsdWUiOiIwIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslIElG IHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8gXCJcXG5cIjtcbkVO RDtcbnRvb2xzLnF1ZXJ5LmFkZChxdWVyeS5maXJzdCwgMSkgJV0iLCJyZXN1bHRz U2F2ZVRvIjoiZmlsZSIsInJlc3VsdHNGaWxlTmFtZSI6IlslIElGIHAxLmluZm8u c3VjY2VzcyA9PSAxO1xuXHRVU0UgTWF0aDtcblx0J3Rlc3RfcmF3L3VzXycgXyBN YXRoLmludChxdWVyeS5udW0gLyAyMDAwKSBfICcvJyBfIHF1ZXJ5LmdsIF8gJ18n IF8gcXVlcnkuaGwgXyAnXycgXyBxdWVyeS5pZCBfICdfJyBfIHF1ZXJ5Lmt3IF8g Jy5odG1sJztcbkVORCAlXSIsImFkZGl0aW9uYWxGb3JtYXRzIjpbWyJ0ZXN0X2Zh aWwvZmFpbGVkLnR4dCIsIlslIElGIHAxLmluZm8uc3VjY2VzcyA9PSAwO1xuXHRx dWVyeS5vcmlnIF8gXCJcXG5cIjtcbkVORCAlXSJdLFsiJGRhdGVmaWxlLmZvcm1h dCgpLnR4dCIsIiRwMS5wcmVzZXQiXV0sInJlc3VsdHNVbmlxdWUiOiJubyIsInF1 ZXJ5Rm9ybWF0IjpbIlslIHEgPSBxdWVyeS5rdyB8IHVyaTtcbklGIHF1ZXJ5Lmx2 bCA9PSAwO1xuXHQnaHR0cHM6Ly8nIF8gcXVlcnkudGxkIF8gJy9zZWFyY2g/cT0n IF8gcSBfICcmcHdzPTAmZ2w9JyBfIHF1ZXJ5LmdsIF8gJyZobD0nIF8gcXVlcnku aGw7XG5FTFNJRiBxdWVyeS5sdmwgPT0gMTtcblx0J2h0dHBzOi8vJyBfIHF1ZXJ5 LnRsZCBfICcvc2VhcmNoP3E9JyBfIHEgXyAnJmdsPScgXyBxdWVyeS5nbCBfICcm aGw9JyBfIHF1ZXJ5LmhsIF8gJyZzdGFydD0xMCZudW09MTAwJztcbkVORCAlXSJd LCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZlRmFpbGVkUXVlcmllcyI6ZmFsc2Us Iml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExldmVscyI6dHJ1ZSwicXVlcnlCdWls ZGVyc0FmdGVySXRlcmF0b3IiOmZhbHNlLCJxdWVyeUJ1aWxkZXJzT25BbGxMZXZl bHMiOnRydWV9LCJyZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZhbHNlfSwi ZG9Mb2ciOiJubyIsImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25zIjpmYWxz ZSwicmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwicXVlcnlC dWlsZGVycyI6W3sic291cmNlIjoicXVlcnkiLCJ0eXBlIjoic3RyaW5nU3BsaXQi LCJzZXBhcmF0b3IiOiI7IiwidG8iOlsidGxkIiwiZ2wiLCJobCIsImlkIiwia3ci XX1dLCJyZXN1bHRzQnVpbGRlcnMiOltdLCJjb25maWdPdmVycmlkZXMiOltdLCJy dW5UYXNrT25Db21wbGV0ZSI6bnVsbCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNG aWxlIjpmYWxzZSwicnVuVGFza09uQ29tcGxldGVDb25maWciOiJkZWZhdWx0Iiwi dG9vbHNKUyI6IiJ9fQ==
Thanks! Never would have figured that out. Saved the day again. Thanks for all of your help and patience!
Ugh, another quirky issue cropped up. I have it set to so that every 2000 queries a new directory is created to keep the files per directory under control. The problem is sometimes the two queries per keyword are saved in two different files in different directories. Is there any way to ensure just a single file per query is created? If that is not possible how would you set it up to save each query to a separate file?
This occurs because the 1st page can be processed, for example, under the number 1990, and the second - under 2057. Therefore, in the current configuration file may be located in different folders. There is a solution. It is necessary to generate name of the folder on the first page and send it to the second, there to be used. So here are working preset: Spoiler: Code Code: eyJwcmVzZXQiOiJkZWZhdWx0IiwidmFsdWUiOnsicHJlc2V0IjoiZGVmYXVsdCIs InBhcnNlcnMiOltbIk5ldDo6SFRUUCIsImRlZmF1bHQiLHsidHlwZSI6Im92ZXJy aWRlIiwiaWQiOiJnb29kQ29kZSIsInZhbHVlIjpbMjAwXX0seyJ0eXBlIjoib3Zl cnJpZGUiLCJpZCI6InVzZXItYWdlbnQiLCJ2YWx1ZSI6Ik1vemlsbGEvNS4wIChX aW5kb3dzIE5UIDYuMTsgV09XNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUws IGxpa2UgR2Vja28pIENocm9tZS80Ny4wLjI1MjYuMTExIFNhZmFyaS81MzcuMzYi fSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHlyZXRyaWVzIiwidmFsdWUi OiIxMDAifSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicHJveHliYW5uZWRjbGVh bnVwIiwidmFsdWUiOiIwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InRpbWVv dXQiLCJ2YWx1ZSI6IjMwIn0seyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6InVzZXBy b3h5IiwidmFsdWUiOmZhbHNlfSx7InR5cGUiOiJvdmVycmlkZSIsImlkIjoicmVx dWVzdGRlbGF5IiwidmFsdWUiOiIwIn1dXSwicmVzdWx0c0Zvcm1hdCI6IlslIElG IHAxLmluZm8uc3VjY2VzcyA9PSAxO1xuXHRwMS5kYXRhIF8gXCJcXG5cIjtcbkVO RDtcblVTRSBNYXRoO1xuZGlyID0gTWF0aC5pbnQocXVlcnkubnVtIC8gMjAwMCk7 XG50b29scy5xdWVyeS5hZGQocXVlcnkuZmlyc3QgXyAnOycgXyBkaXIsIDEpICVd IiwicmVzdWx0c1NhdmVUbyI6ImZpbGUiLCJyZXN1bHRzRmlsZU5hbWUiOiJbJSBJ RiBwMS5pbmZvLnN1Y2Nlc3MgPT0gMTtcblx0SUYgcXVlcnkubHZsID09IDA7XG4g ICAgXHRVU0UgTWF0aDtcbiAgICAgICAgZGlyID0gTWF0aC5pbnQocXVlcnkubnVt IC8gMjAwMCk7XG4gICAgRUxTSUYgcXVlcnkubHZsID09IDE7XG4gICAgXHRkaXIg PSBxdWVyeS5kaXI7XG4gICAgRU5EO1xuXHQndGVzdF9yYXcvdXNfJyBfIGRpciBf ICcvJyBfIHF1ZXJ5LmdsIF8gJ18nIF8gcXVlcnkuaGwgXyAnXycgXyBxdWVyeS5p ZCBfICdfJyBfIHF1ZXJ5Lmt3IF8gJy5odG1sJztcbkVORCAlXSIsImFkZGl0aW9u YWxGb3JtYXRzIjpbWyJ0ZXN0X2ZhaWwvZmFpbGVkLnR4dCIsIlslIElGIHAxLmlu Zm8uc3VjY2VzcyA9PSAwO1xuXHRxdWVyeS5vcmlnIF8gXCJcXG5cIjtcbkVORCAl XSJdLFsiJGRhdGVmaWxlLmZvcm1hdCgpLnR4dCIsIiRwMS5wcmVzZXQiXV0sInJl c3VsdHNVbmlxdWUiOiJubyIsInF1ZXJ5Rm9ybWF0IjpbIlslIHEgPSBxdWVyeS5r dyB8IHVyaTtcbklGIHF1ZXJ5Lmx2bCA9PSAwO1xuXHQnaHR0cHM6Ly8nIF8gcXVl cnkudGxkIF8gJy9zZWFyY2g/cT0nIF8gcSBfICcmcHdzPTAmZ2w9JyBfIHF1ZXJ5 LmdsIF8gJyZobD0nIF8gcXVlcnkuaGw7XG5FTFNJRiBxdWVyeS5sdmwgPT0gMTtc blx0J2h0dHBzOi8vJyBfIHF1ZXJ5LnRsZCBfICcvc2VhcmNoP3E9JyBfIHEgXyAn JmdsPScgXyBxdWVyeS5nbCBfICcmaGw9JyBfIHF1ZXJ5LmhsIF8gJyZzdGFydD0x MCZudW09MTAwJztcbkVORCAlXSJdLCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZl RmFpbGVkUXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExl dmVscyI6dHJ1ZSwicXVlcnlCdWlsZGVyc0FmdGVySXRlcmF0b3IiOmZhbHNlLCJx dWVyeUJ1aWxkZXJzT25BbGxMZXZlbHMiOnRydWV9LCJyZXN1bHRzT3B0aW9ucyI6 eyJvdmVyd3JpdGUiOmZhbHNlfSwiZG9Mb2ciOiJubyIsImtlZXBVbmlxdWUiOiJO byIsIm1vcmVPcHRpb25zIjpmYWxzZSwicmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1 bHRzQXBwZW5kIjoiIiwicXVlcnlCdWlsZGVycyI6W3sic291cmNlIjoicXVlcnki LCJ0eXBlIjoic3RyaW5nU3BsaXQiLCJzZXBhcmF0b3IiOiI7IiwidG8iOlsidGxk IiwiZ2wiLCJobCIsImlkIiwia3ciLCJkaXIiXX1dLCJyZXN1bHRzQnVpbGRlcnMi OltdLCJjb25maWdPdmVycmlkZXMiOltdLCJydW5UYXNrT25Db21wbGV0ZSI6bnVs bCwidXNlUmVzdWx0c0ZpbGVBc1F1ZXJpZXNGaWxlIjpmYWxzZSwicnVuVGFza09u Q29tcGxldGVDb25maWciOiJkZWZhdWx0IiwidG9vbHNKUyI6IiJ9fQ==
