SE::Yandex::Suggest - Yandex Search Suggest Scraper
Overview of the scraper
A scraper for search suggestions by keywords in Yandex. Thanks to the SE::Yandex::Suggest scraper, you can automatically collect keyword bases from the search engine Yandex's suggestions based on a query. Using the SE::Yandex::Suggest scraper, you can quickly and easily scrape Yandex suggestions for a query based on the region.
The Yandex suggestions scraper solves one of the main SEO tasks, namely the rapid automated acquisition of an expanded semantic core. Google search suggestions allow covering the maximum number of phrases, and in conjunction with the Yandex Keyword scraper - 
SE::Yandex::WordStat you will get the most complete semantics, which will help attract more organic traffic.
Thanks to A-Parser's multi-threaded operation, query processing speed can reach 5700 queries per minute, which on average allows receiving up to 17000-18000 results per minute.
You can use automatic query multiplication, substitution of subqueries from files, iteration through alphanumeric combinations and lists to get the maximum possible number of results. By using result filtering you can immediately clean up the result by removing all unnecessary junk (using negative keywords).
A-Parser functionality allows saving the parsing settings of the SE::Yandex::Suggest scraper for future use (presets), , setting up a parsing schedule, and much more.
Result saving is possible in the form and structure you need, thanks to the built-in powerful templater Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.
Collected data
- Number of results per query
- Suggestions for the query
- When choosing the 2nd query method, additional data and the type of suggestions are also collected
Use Cases
- Collecting keyword bases
Queries
As queries, you need to specify search phrases, for example:
forex
write essay
essays online
buy a car in Moscow
news
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add a list of other words to each query, let's specify a few main queries:
essay
article
thesis
In the query format, we will specify a macro for substituting additional words from the file Keywords.txt, this method allows increasing query variability manifold:
{subs:Keywords} $query
This macro will create as many additional queries as there are in the file for each initial search query, which in total will give [number of initial queries (domains)] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro's operation.
For example, if the file Keywords.txt will contain:
buy
cheap
As a result, the substitution macro will turn 3 main queries into 6:
buy essay
cheap essay
buy article
cheap article
buy thesis
cheap thesis
Output results examples
A-Parser supports flexible result formatting thanks to the built-in templater Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Suggests list output
Result format:
$results.format('$suggest\n')
Result example:
essays online free
essays online order
essays online free download
essays online download
essays online make
essays online do
essays online find
essays online make free
essays online order
Output of the query, the count of found suggestions, and the suggestions for the query
Result format:
$query - $totalcount:\n$results.format('$suggest\n')
Result example:
essays online - 10:
essays online
essays online free
essays online order
essays online free download
essays online download
essays online make
essays online do
essays online find
essays online make free
essays online order
Output to CSV table
The built-in utility $tools.CSVLine allows creating correct tabular documents ready for import into Excel or Google Sheets.
Result format:
[% FOREACH i IN results;
tools.CSVline(i.suggest);
END %]
File name:
$datefile.format().csv
Initial text:
Suggestions
In the General result format, the Template Toolkit templater is used to output the suggest elements of the results array in a FOREACH loop.
For the file name of the results, you just need to change the file extension to csv.
For the "Initial text" option to be available in the Task Editor, , you need to activate "More options". In "Initial text", we write the column names separated by commas and leave the second line empty.
Keyword competition
Saving in SQL format
Result format:
[% FOREACH results;
"INSERT INTO serp VALUES('" _ query _ "', '"; suggest _ "')\n";
END %]
Result example:
INSERT INTO serp VALUES('term papers online', 'term papers online')
INSERT INTO serp VALUES('term papers online', 'free term papers online')
INSERT INTO serp VALUES('term papers online', 'order term papers online')
INSERT INTO serp VALUES('term papers online', 'download free term papers online')
INSERT INTO serp VALUES('term papers online', 'download term papers online')
INSERT INTO serp VALUES('term papers online', 'create term papers online')
INSERT INTO serp VALUES('term papers online', 'do term papers online')
INSERT INTO serp VALUES('term papers online', 'find term papers online')
...
Dump results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.totalcount = p1.totalcount;
obj.suggests = [];
FOREACH item IN p1.serp;
obj.suggests.push(item.suggest);
END;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Result example:
[{"suggests":["essays online","essays online free","essays online order","essays online free download","essays online download","essays online make","essays online do","essays online find","essays online make free","essays online order"],"totalcount":10}]
For the "Initial text" and "Final text" options to be available in the Task Editor, , you need to activate "More options".
Results processing
A-Parser allows processing results directly during parsing, in this section we have presented the most popular use cases for the SE::Yandex::Suggest scraper
Parse to level option
This option tells the scraper to substitute the received results into the query queue to the specified depth level, for example:
- If level 1 is specified, the scraper will add all results obtained from the initial query to the queries
- If level 2 is specified, the scraper will add all results obtained from the initial query to the queries + all results obtained for queries from level 1
- and so on
In simple words - this is the substitution of received results into the query queue, which in turn allows scraping more results. Since there will likely be duplicates in the results, it is recommended to enable unique queries
Download example
How to import the example into A-Parser
eJx1VFtv2jAU/iuVhdRVQggKfVjeKCrSJtZ0pX2YgAe3Pom8OrZnO6woyn/vsRPi
wLoXy+f2nct37Io4at/sgwELzpJkUxEd7iQhDDJaCkeGRFNjwXjzhqzvkuQXlQze
k2Rd5jlY79H5VsQdNGC02oMxnAEaOUM5U6agDrEbyD0VpXcbNBo7auxfLge2Ad1u
5eUVqXuA2nEl7REv1LSCPYiINhnX/y9AmOh4PZnWu92QtMmXIbevRk9Gbf+dcU33
8KR8B1xAVC9RuqdF6IFRB956bOJq5N49AmWM+6KpaDL4Ccasz5L/CTOQCn3xajjY
pVEFqhwEAK88HKvbkEGQCUKUIfZnE0OSjAoLQ2Kx1CXFQti5hTsw1CmTtkNMKqLk
XIgwwOgW8G9LLhjSPc8w6Fsb+LlL+g9G3bXXT4VU/DVYQ4cSpNv0R4xiaqXy4zAE
L7hD2S5UKT0xY1S+AehuZvferVAGujQtcpsdN1qD9LxHyuY6qk7aOKHlVPmqZMbz
tN2lo2cpn/DZpHKhCi3A9yVLIZAWC49xPea2pcELscDz4EVIcfLgnFLCfl83pWrD
cf1ufIEFTrKftYV8pUI8P676Fh9ondKpvDPmjL6wGGRbjmezsT+nN+E+690bfThn
1+F8uQgqCCcL50t0mn5t9TIwewHW0gPxlTnIFe4szq3edT9J999Un/4nSVXjRvy2
D423H5/3RR3yYJFufOr1Bxh5omw=
Filtering results (using negative keywords)
Using negative keywords, it is immediately possible to remove results that you do not need. Similarly, using a filter, you can only keep those results that contain the desired words.
Download example
How to import the example into A-Parser
eJx1VN9vGjEM/leqCKmrVCEoIG33RlGRNrHSFfowAQ+BMyxrLrklOVbE7n+f44S7
g3UvUWx/tr/4R47McftqnwxYcJYliyPL6c4SlsKWF9KxW5ZzY8F484LNHpLkO1cp
vCXJrNjtwHpEhT0yd8gBvfUejBEpoFGkKG+1ybjD2CHknsvCw1pBY9vB/uG6ZUPQ
5VJd37CyETB3Qit7ikecJrAHWUfrdsr/E5CmBt51ew3kVkgH3hrZJYt488kiHba6
jbh5cDKgtEPGmx+N1yyLTr/f92fvI51r0mzovvlDQpeEAd07DWi4rwnUu2sY+rVD
z78mFAKzsXK1OpG2Y6qfr2jebcceVsYZ38Nch6dCrR6j9Mgz6kPKHXjrqRE3bffm
I/A0FT4flyGDn4I664sSv+jlSiMWr0aAHRudocoBBfDKw4ndgrVI9uUsyPdb8GHJ
lksLWG+kOuZIJL20CCw+d9pM4yAkR6bVUEoaghpG8e8LIVMc2eEWnT5Hx/ch039i
lNXzmqlwnH4b5FBFIel++rX2SvVE707FkCITDmU70oXyjemg8hUgr2r26GGZNlCl
iZFjdtzKHJSf3bplw7xWnT3jrC3nyo1WW7Gbxn04IQs1x9WfqpHOcgn+XaqQEtti
4bkej6GNbfBCTfDSeUQpzj4Np7W0X2aBam4Ejt/AE8ywks2sMeSGS/nyPGlavKN1
Op+qB2Mu2keDwRpLNGjsXnO56OzTPvXXV2GJ6EzrBY279ynqFXX2CqzlB+aZOdhp
nFmsW7mqfsPqzzy++ycmxxIn4qd9CmhfPo9FHfbB0gJ3y7/kMN7o
See also: Result Filters
Possible settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Region | Russia | Selection of the region from which the search is performed |
| Request method | Method 1 | Query method: Method 1 - old method, same as before; Method 2 - new method, allows collecting all expanded suggestions (ads, facts, links to sites) |