How to make your parser of suggests? Examples for Baidu, DuckDuckGo, GooglePlay

For example, take baidu.com. For other services, an algorithm will be about the same.

First you need to find URL, which returns suggestions as you type in the search box. For this I use the developer tools in Chrome.

• Open http://www.baidu.com/ in browser.
• Writing any request, for example test
• Open the Developer Tools (Ctrl+Shift+I), switch to the Network tab.
• Put the cursor in the search box, it is possible to add some character (for example space). It is necessary that displays suggests.
• In the Tools see sent requests. We are looking for the one that returns a suggests.
  
• On the screenshot highlighted just what we need (the first request). I recommend for convenience to remove the excess parameters from the query, experiment with it in a browser. But it should be done with extreme care, because happens that by removing some of the parameters, the result is quite different. In the example, the query looks like this:
  
  Code:

  http://suggestion.baidu.com/su?wd=test&sugmode=2&json=1&p=3&sid=18881_18728_1431_18241_18878_12825_18730_18767_18777_18546_18779_17000_18782_17073_15729_11820_17999_10633&req=2&bs=test&pbs=test&csor=4&cb=jQuery11020484825603896752_1452509913872&_=1452509913893

• After cutting it will look like this:
  
  Code:

  http://suggestion.baidu.com/su?wd=test&json=1&req=2&bs=test&pbs=test

Result after sending such a request would look like this:

In fact it is JSON, but enclosed in parentheses.

So, with a request, and knowing how the result looks, make a preset on the basis of Net::HTTP, with parsing JSON and output only suggests using a template engine capabilities. Also, using tools.query.add can implement substitution obtained suggests back to queries, i.e. parse in depth (in examle - 2).

Spoiler: Code for import

Code:

eyJwcmVzZXQiOiJCYWlkdSBzdWdnZXN0cyIsInZhbHVlIjp7InByZXNldCI6IkJh
aWR1IHN1Z2dlc3RzIiwicGFyc2VycyI6W1siTmV0OjpIVFRQIiwiZGVmYXVsdCIs
eyJ0eXBlIjoib3ZlcnJpZGUiLCJpZCI6Imdvb2RDb2RlIiwidmFsdWUiOjIwMH0s
eyJ0eXBlIjoiY3VzdG9tUmVzdWx0IiwicmVzdWx0IjoiZGF0YSIsInJlZ2V4Ijoi
XFwoKC4rPylcXCk7IiwicmVnZXhUeXBlIjoiIiwicmVzdWx0VHlwZSI6ImZsYXQi
LCJhcnJheU5hbWUiOiIiLCJyZXN1bHRzIjpbImpzb25kYXRhIl19XV0sInJlc3Vs
dHNGb3JtYXQiOiJbJSBsZXZlbCA9IDI7ICNcdTA0MTNcdTA0M2JcdTA0NDNcdTA0
MzFcdTA0MzhcdTA0M2RcdTA0MzAgXHUwNDNmXHUwNDNlXHUwNDM0XHUwNDQxXHUw
NDQyXHUwNDMwXHUwNDNkXHUwNDNlXHUwNDMyXHUwNDNhXHUwNDM4IFx1MDQzZlx1
MDQzZVx1MDQzNFx1MDQ0MVx1MDQzYVx1MDQzMFx1MDQzN1x1MDQzZVx1MDQzYSwg
MCBcdTA0MzRcdTA0M2JcdTA0NGYgXHUwNDNlXHUwNDQyXHUwNDNhXHUwNDNiXHUw
NDRlXHUwNDQ3XHUwNDM1XHUwNDNkXHUwNDM4XHUwNDRmXG5cbiNcdTA0MTJcdTA0
NGJcdTA0MzJcdTA0M2VcdTA0MzQgXHUwNDNmXHUwNDNlXHUwNDM0XHUwNDQxXHUw
NDNhXHUwNDMwXHUwNDM3XHUwNDNlXHUwNDNhXG5qc29uID0gdG9vbHMucGFyc2VK
U09OKHAxLmpzb25kYXRhKTtcbklGIGpzb24ucy5zaXplID4gMDtcblx0anNvbi5z
LmpvaW4oXCJcXG5cIikgXyBcIlxcblwiO1xuRU5EO1xuXG4jXHUwNDFmXHUwNDNl
XHUwNDM0XHUwNDQxXHUwNDQyXHUwNDMwXHUwNDNkXHUwNDNlXHUwNDMyXHUwNDNh
XHUwNDMwIFx1MDQzZlx1MDQzZVx1MDQzNFx1MDQ0MVx1MDQzYVx1MDQzMFx1MDQz
N1x1MDQzZVx1MDQzYSBcdTA0MzIgXHUwNDM3XHUwNDMwXHUwNDNmXHUwNDQwXHUw
NDNlXHUwNDQxXHUwNDRiXG5GT1JFQUNIIHN1ZyBJTiBqc29uLnM7XG5cdHRvb2xz
LnF1ZXJ5LmFkZChzdWcsIGxldmVsKTtcbkVORCAlXSIsInJlc3VsdHNTYXZlVG8i
OiJmaWxlIiwicmVzdWx0c0ZpbGVOYW1lIjoiJGRhdGVmaWxlLmZvcm1hdCgpLnR4
dCIsImFkZGl0aW9uYWxGb3JtYXRzIjpbXSwicmVzdWx0c1VuaXF1ZSI6Im5vIiwi
cXVlcnlGb3JtYXQiOlsiaHR0cDovL3N1Z2dlc3Rpb24uYmFpZHUuY29tL3N1P3dk
PSRxdWVyeSZqc29uPTEmcmVxPTImYnM9JHF1ZXJ5JnBicz0kcXVlcnkiXSwidW5p
cXVlUXVlcmllcyI6ZmFsc2UsInNhdmVGYWlsZWRRdWVyaWVzIjpmYWxzZSwiaXRl
cmF0b3JPcHRpb25zIjp7Im9uQWxsTGV2ZWxzIjp0cnVlLCJxdWVyeUJ1aWxkZXJz
QWZ0ZXJJdGVyYXRvciI6ZmFsc2UsInF1ZXJ5QnVpbGRlcnNPbkFsbExldmVscyI6
ZmFsc2V9LCJyZXN1bHRzT3B0aW9ucyI6eyJvdmVyd3JpdGUiOmZhbHNlfSwiZG9M
b2ciOiJubyIsImtlZXBVbmlxdWUiOiJObyIsIm1vcmVPcHRpb25zIjpmYWxzZSwi
cmVzdWx0c1ByZXBlbmQiOiIiLCJyZXN1bHRzQXBwZW5kIjoiIiwicXVlcnlCdWls
ZGVycyI6W10sInJlc3VsdHNCdWlsZGVycyI6W10sImNvbmZpZ092ZXJyaWRlcyI6
W119fQ==

Example of result:

 

Suggests from duckduckgo.com

For suggests from duckduckgo.com query has this form:

Code:

https://ac.duckduckgo.com/ac/?callback=autocompleteCallback&q=test&_=1452761605181

After cutting:

Code:

https://ac.duckduckgo.com/ac/?q=test

Spoiler: Code for import

Code:

eyJwcmVzZXQiOiJEdWNrRHVja0dvIHN1Z2dlc3RzIiwidmFsdWUiOnsicHJlc2V0
IjoiRHVja0R1Y2tHbyBzdWdnZXN0cyIsInBhcnNlcnMiOltbIk5ldDo6SFRUUCIs
ImRlZmF1bHQiLHsidHlwZSI6Im92ZXJyaWRlIiwiaWQiOiJnb29kQ29kZSIsInZh
bHVlIjoyMDB9XV0sInJlc3VsdHNGb3JtYXQiOiJbJSBsZXZlbCA9IDI7ICNcdTA0
MTNcdTA0M2JcdTA0NDNcdTA0MzFcdTA0MzhcdTA0M2RcdTA0MzAgXHUwNDNmXHUw
NDNlXHUwNDM0XHUwNDQxXHUwNDQyXHUwNDMwXHUwNDNkXHUwNDNlXHUwNDMyXHUw
NDNhXHUwNDM4IFx1MDQzZlx1MDQzZVx1MDQzNFx1MDQ0MVx1MDQzYVx1MDQzMFx1
MDQzN1x1MDQzZVx1MDQzYSwgMCBcdTA0MzRcdTA0M2JcdTA0NGYgXHUwNDNlXHUw
NDQyXHUwNDNhXHUwNDNiXHUwNDRlXHUwNDQ3XHUwNDM1XHUwNDNkXHUwNDM4XHUw
NDRmXG5cbiNcdTA0MTJcdTA0NGJcdTA0MzJcdTA0M2VcdTA0MzQgXHUwNDNmXHUw
NDNlXHUwNDM0XHUwNDQxXHUwNDNhXHUwNDMwXHUwNDM3XHUwNDNlXHUwNDNhXG5q
c29uID0gdG9vbHMucGFyc2VKU09OKHAxLmRhdGEpO1xuRk9SRUFDSCBqc29uO1xu
XHRwaHJhc2UgXyBcIlxcblwiO1xuRU5EO1xuXG4jXHUwNDFmXHUwNDNlXHUwNDM0
XHUwNDQxXHUwNDQyXHUwNDMwXHUwNDNkXHUwNDNlXHUwNDMyXHUwNDNhXHUwNDMw
IFx1MDQzZlx1MDQzZVx1MDQzNFx1MDQ0MVx1MDQzYVx1MDQzMFx1MDQzN1x1MDQz
ZVx1MDQzYSBcdTA0MzIgXHUwNDM3XHUwNDMwXHUwNDNmXHUwNDQwXHUwNDNlXHUw
NDQxXHUwNDRiXG5GT1JFQUNIIGpzb247XG5cdHRvb2xzLnF1ZXJ5LmFkZChwaHJh
c2UsIGxldmVsKTtcbkVORCAlXSIsInJlc3VsdHNTYXZlVG8iOiJmaWxlIiwicmVz
dWx0c0ZpbGVOYW1lIjoiJGRhdGVmaWxlLmZvcm1hdCgpLnR4dCIsImFkZGl0aW9u
YWxGb3JtYXRzIjpbXSwicmVzdWx0c1VuaXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQi
OlsiaHR0cHM6Ly9hYy5kdWNrZHVja2dvLmNvbS9hYy8/cT0kcXVlcnkiXSwidW5p
cXVlUXVlcmllcyI6dHJ1ZSwic2F2ZUZhaWxlZFF1ZXJpZXMiOmZhbHNlLCJpdGVy
YXRvck9wdGlvbnMiOnsib25BbGxMZXZlbHMiOnRydWUsInF1ZXJ5QnVpbGRlcnNB
ZnRlckl0ZXJhdG9yIjpmYWxzZSwicXVlcnlCdWlsZGVyc09uQWxsTGV2ZWxzIjpm
YWxzZX0sInJlc3VsdHNPcHRpb25zIjp7Im92ZXJ3cml0ZSI6ZmFsc2V9LCJkb0xv
ZyI6Im5vIiwia2VlcFVuaXF1ZSI6Ik5vIiwibW9yZU9wdGlvbnMiOmZhbHNlLCJy
ZXN1bHRzUHJlcGVuZCI6IiIsInJlc3VsdHNBcHBlbmQiOiIiLCJxdWVyeUJ1aWxk
ZXJzIjpbXSwicmVzdWx0c0J1aWxkZXJzIjpbXSwiY29uZmlnT3ZlcnJpZGVzIjpb
XX19

 

Suggests from Google Play

Spoiler: Code for import

Code:

eyJwcmVzZXQiOiJHUCBzdWdnZXN0cyIsInZhbHVlIjp7InByZXNldCI6IkdQIHN1
Z2dlc3RzIiwicGFyc2VycyI6W1siTmV0OjpIVFRQIiwiZGVmYXVsdCIseyJ0eXBl
Ijoib3ZlcnJpZGUiLCJpZCI6Imdvb2RDb2RlIiwidmFsdWUiOlsyMDBdfV1dLCJy
ZXN1bHRzRm9ybWF0IjoiWyUgbGV2ZWwgPSAyOyAjXHUwNDEzXHUwNDNiXHUwNDQz
XHUwNDMxXHUwNDM4XHUwNDNkXHUwNDMwIFx1MDQzZlx1MDQzZVx1MDQzNFx1MDQ0
MVx1MDQ0Mlx1MDQzMFx1MDQzZFx1MDQzZVx1MDQzMlx1MDQzYVx1MDQzOCBcdTA0
M2ZcdTA0M2VcdTA0MzRcdTA0NDFcdTA0M2FcdTA0MzBcdTA0MzdcdTA0M2VcdTA0
M2EsIDAgXHUwNDM0XHUwNDNiXHUwNDRmIFx1MDQzZVx1MDQ0Mlx1MDQzYVx1MDQz
Ylx1MDQ0ZVx1MDQ0N1x1MDQzNVx1MDQzZFx1MDQzOFx1MDQ0ZlxuXG4jXHUwNDEy
XHUwNDRiXHUwNDMyXHUwNDNlXHUwNDM0IFx1MDQzZlx1MDQzZVx1MDQzNFx1MDQ0
MVx1MDQzYVx1MDQzMFx1MDQzN1x1MDQzZVx1MDQzYVxuanNvbiA9IHRvb2xzLnBh
cnNlSlNPTihwMS5kYXRhKTtcbkZPUkVBQ0gganNvbjtcblx0cyBfIFwiXFxuXCI7
XG5FTkQ7XG5cbiNcdTA0MWZcdTA0M2VcdTA0MzRcdTA0NDFcdTA0NDJcdTA0MzBc
dTA0M2RcdTA0M2VcdTA0MzJcdTA0M2FcdTA0MzAgXHUwNDNmXHUwNDNlXHUwNDM0
XHUwNDQxXHUwNDNhXHUwNDMwXHUwNDM3XHUwNDNlXHUwNDNhIFx1MDQzMiBcdTA0
MzdcdTA0MzBcdTA0M2ZcdTA0NDBcdTA0M2VcdTA0NDFcdTA0NGJcbkZPUkVBQ0gg
anNvbjtcblx0dG9vbHMucXVlcnkuYWRkKHMsIGxldmVsKTtcbkVORCAlXSIsInJl
c3VsdHNTYXZlVG8iOiJmaWxlIiwicmVzdWx0c0ZpbGVOYW1lIjoiJGRhdGVmaWxl
LmZvcm1hdCgpLnR4dCIsImFkZGl0aW9uYWxGb3JtYXRzIjpbXSwicmVzdWx0c1Vu
aXF1ZSI6Im5vIiwicXVlcnlGb3JtYXQiOlsiaHR0cHM6Ly9tYXJrZXQuYW5kcm9p
ZC5jb20vc3VnZ2VzdC9TdWdnUmVxdWVzdD9qc29uPTEmYz0wJnF1ZXJ5PSRxdWVy
eSZobD1lbiZnbD1VUyJdLCJ1bmlxdWVRdWVyaWVzIjp0cnVlLCJzYXZlRmFpbGVk
UXVlcmllcyI6ZmFsc2UsIml0ZXJhdG9yT3B0aW9ucyI6eyJvbkFsbExldmVscyI6
dHJ1ZSwicXVlcnlCdWlsZGVyc0FmdGVySXRlcmF0b3IiOmZhbHNlLCJxdWVyeUJ1
aWxkZXJzT25BbGxMZXZlbHMiOmZhbHNlfSwicmVzdWx0c09wdGlvbnMiOnsib3Zl
cndyaXRlIjpmYWxzZX0sImRvTG9nIjoibm8iLCJrZWVwVW5pcXVlIjoiTm8iLCJt
b3JlT3B0aW9ucyI6ZmFsc2UsInJlc3VsdHNQcmVwZW5kIjoiIiwicmVzdWx0c0Fw
cGVuZCI6IiIsInF1ZXJ5QnVpbGRlcnMiOltdLCJyZXN1bHRzQnVpbGRlcnMiOltd
LCJjb25maWdPdmVycmlkZXMiOltdfX0=

• In the General query format, you can change parameters hl= and gl=, thus changing the language and country respectively.