SE::Yandex::Direct::Frequency - Frequency Check
Scraper Overview
Search query scraper from Yandex Direct. The keyword scraper for Yandex Direct can automatically collect the number of impressions per query for a specified period, scraping Yandex Direct search suggestions.
You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results. Using result filtering you can immediately clean up the result by removing all unnecessary junk (using negative keywords).
A-Parser functionality allows you to save scraper settings SE::Yandex::Direct::Frequency for future use (presets), set a scraping schedule, and much more.
Saving results is possible in the form and structure that you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
Collected Data
- Number of impressions per query for a specified period
- Scraping of additional keywords (suggestions)
Capabilities
- Supports selection of search region (with subgroups)
- Supports selection of period (last month, specific month, quarter, year) (only when scraping with accounts)
Use Cases
- Estimating the amount of traffic by keyword (frequency)
Queries
When scraping with accounts, scraping occurs in "batches" of 10 keywords per request. As queries, you need to specify keywords, just as if you were entering them directly into the Yandex Direct search form, for example:
_тест
"!город"
"город"_
Query Substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add some list of other words to each query, let's specify several main queries:
essay
article
thesis
In the query format, specify the macro for substituting additional words from the file Keywords.txt, this method allows you to increase the variability of queries many times over:
{subs:Keywords} $query
This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries(domains)] x [number of queries in the file Keywords] = [total number of queries] as a result of the macro's work.
For example, if the file Keywords.txt will contain:
buy
cheap
As a result, the substitution macro will turn 3 main queries into 6:
buy essay
cheap essay
buy article
cheap article
buy thesis
cheap thesis
Output Results Examples
A-Parser supports flexible formatting of results thanks to the built-in templating engine Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Export of keyword frequency and list of suggestions
Result format:
$query: $views\n$suggests.format('$key\n')
Example of result:
рефераты онлайн: 2771
реферат технология
реферат скачивать тема
обучение реферат
заказывать реферат
бесплатный реферат тема
английский реферат язык
бесплатный реферат скачивать тема
сочинение тест
находить реферат
реферат тест
видео реферат
переводчик реферат
находить работа реферат
реферати тема
Output of keyword frequency
Result format:
$query: $views\n
Example of result:
рефераты онлайн: 2771
Output in CSV Table
The built-in utility $tools.CSVLine allows you to create correct table documents, ready for import into Excel or Google Sheets.
Result format:
[% FOREACH i IN suggests; tools.CSVline(i.key); END %]
File name:
$datefile.format().csv
Initial text:
Подсказки
In the General result format, the Template Toolkit templating engine is used to output the elements of the key array in the FOREACH loop suggests.
In the file name of the results, you just need to change the file extension to csv.
To make the "Initial text" option available in the Task Editor, you need to activate "More options". In "Initial text," write the column names separated by commas and make the second line empty.
Saving in SQL format
Result format:
[% FOREACH suggests; "INSERT INTO serp VALUES('" _ query _ "', '" _ key _ "')\n"; END %]
Example of result:
INSERT INTO serp VALUES('рефераты онлайн', 'реферат технология')
INSERT INTO serp VALUES('рефераты онлайн', 'реферат скачивать тема')
INSERT INTO serp VALUES('рефераты онлайн', 'обучение реферат')
INSERT INTO serp VALUES('рефераты онлайн', 'заказывать реферат')
INSERT INTO serp VALUES('рефераты онлайн', 'бесплатный реферат тема')
INSERT INTO serp VALUES('рефераты онлайн', 'английский реферат язык')
INSERT INTO serp VALUES('рефераты онлайн', 'бесплатный реферат скачивать тема')
INSERT INTO serp VALUES('рефераты онлайн', 'сочинение тест')
INSERT INTO serp VALUES('рефераты онлайн', 'находить реферат')
INSERT INTO serp VALUES('рефераты онлайн', 'реферат тест')
INSERT INTO serp VALUES('рефераты онлайн', 'видео реферат')
INSERT INTO serp VALUES('рефераты онлайн', 'переводчик реферат')
INSERT INTO serp VALUES('рефераты онлайн', 'находить работа реферат')
INSERT INTO serp VALUES('рефераты онлайн', 'реферати тема')
...
Dumping results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.totalcount = p1.totalcount;
obj.suggests = [];
FOREACH item IN p1.results;
obj.suggests.push(item.suggest);
END;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example of result:
[{"suggests":["реферат технология","реферат скачивать тема","обучение реферат","заказывать реферат","бесплатный реферат тема","английский реферат язык","бесплатный реферат скачивать тема","сочинение тест","находить реферат","реферат тест","видео реферат","переводчик реферат","находить работа реферат","реферати тема"],"views":"2771"}]
To make the "Initial text" and "Final text" options available in the Task Editor, you need to activate "More options".
Results processing
A-Parser allows you to process results directly during scraping, in this section we have provided the most popular cases for the scraper SE::Yandex::Direct::Frequency
Selecting a period for a year, quarter, or specific month
To set a period, you need to use the Report period option and select a period from the dropdown list.
Download example
How to import an example into A-Parser
eJyFVG1v0zAQ/iuV1Q8glapj3Qf8rStUApV1rJvQVCrk1Zdi5tie7ZRVUf47d06a
tAPBl5Pv7bnnXpKSRREew7WHADEwviqZS2/G2fID5/fCSHjm/L3ysImczzw8FWA2
+96b3lcVf/TuQfieA6+sZAPmhA/gCWf173SMlZCJQkc2KFncO8CCdgfeKwnoVBL1
FnYndIEB5cHC2R7LUr2kfzdFjjZTaM2qar0eMGwBocPM+lxQK313Nmz6ap1LsYNb
i85MaejMM9SuRE58+lJEIO8wS0CvXg/jMyEIKVVU1ghdV6CGu6p3Rj0RXWYsxuLT
Kwgzb4ljhARAxv2B3Yr1k84Qoki5X+ocxjOhAwxYQKozgUTkS4+K4EW0fuGIT6AZ
WTPReg470F1Ywr8slJa4nUmGSR+bxL+HLP7AqNr2jkvhwn555NCiJO1y8bnLknZu
t4dhaJWriHqY2sLQYkZofARw7cyuKCy3HtoyDXJTHS/VgaET6FY2cZ3ppI2TtZwa
N9ZkartoLu4QWZhb/BwWZmpzp4H6oqPCtQS46c5jEpo1kNIRfJk8TSWQVnvpLFqr
w6dlTdXh7TJ+QQRznORx1QZyI7S+u5kfe1h3Uqh8K0bj8Yjk+UV6j4/etT3J8dsk
H3rJBEnKJB+6oPN3tZ1R4QhbiyeJY6nW7Xfd/ibK/3zdvKxw9T/DdZ1Hc6IstOHA
A+6V8bPqNzmXjXA=
Filtering results (using negative keywords)
By using negative keywords, it is possible to immediately remove the results you do not need. Similarly, using a filter, you can also keep only those results that contain the necessary words.
Download example
How to import an example into A-Parser
eJy9Vltv2jAU/ivIqtRVYhUQUNu8td2QNnWj6+Vhomgy5IR6NXZqO6yI8t937ITY
lJWOauuLsc/lO9+5mHhODNV3+lyBBqNJ3J+TzO1JTC4/xvF3KhJ4iOMPTMHIxHFX
wX0OYjSrva91GTegavBAJxkHUicZVRqURelvdkbbBFKac0Pqc2JmGWA4OQWlWGKB
WILnDBSTCZ6mlOdoMF9KYjKRwtzagE7wQ+QTEjcWi+exxiA9UJ+0Wh0yCMxTlwla
YOqWFJrofDwGjSWpkzuYkUG9NLoqPBQIaSbUjG49LgqThJlHxjmMKX/UGRVMWwOZ
GSYFWpANHFOpELBk4EF3lkz2C4N3uztI6OZG7O6RxWCw5Ky7TmsdsuZ+2cNKeUmn
cCWLTMGLsYPwlU5cmIQasNplmL1982ARqM0J2VNeRLDt9VGvBbt3NLVRTIzRHo+K
ge4qiU0hBhyIFc6WDPtkx51tUXPn/63wIbFROdSJRrZdilySSpFSrlHDsAPUSNVz
BdV2KKQ45vwMpsC9mYM/yRlPcByPU3T6VDr+2aS3hrGoMgxDYcd+KeRQobjTSe+L
90rkmRxj4skQ0+Zswgye9anMhe1Nw40TZFXZvtq5nEgFVZgSuYyOFzMDYcfDd+04
86KVNFY6syocSZGyca8cuaVlLq7w9vfEqbQ32OYlcs6xKxou/IQc67IN9uAJPnU+
dSFs6surTYyUXH++LKhmeFdJ3LEEJ1jJMGoJOaKcX1+chRriJwoPN3mj3W7YNeq4
fTvYF3K3tltuHdacCNyauHXojaKjUi7cL3Uukdunbj0MoEa1LWKLv7et+RiluhOw
hYBzsY8CZun2gZo+1VJx4AGj1loFR2v8olFpJAJWTV+7EjDMogDvbF3B6CBoVsC5
kLeHmzlvE6gZ6JvBBAyfghcZlaGPtgj0Uh0bQb2iIPahn9SV5h2+kkG6VkD6xjX4
1yNYAsIaVLJ5EMM711yZhiTg0AmmvO1B3vxfYZVhK0gnAIlg+9FP1/SdIESQ+Ept
6faBXlnVRjCcEJSl8Z9KX7B5duaI/UoZGEt8vuA3dDGoXr3VE3r+wts3nuNLlfzU
54Wf/ahaL5Th11m7d2Jz8RvgGZAl
See also: Results filters
Possible settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Region | Russia | Search region |
| Use Yandex accounts | ☐ | Determines whether to use Yandex accounts |
| AntiGate preset | default | Preset for the scraper Util::AntiGate. It is necessary to preconfigure the scraper  Util::AntiGate - specify your access key and other parameters, then select the created preset here |
| Report period | Month: last 30 days | Selection of the period (last month, specific month, quarter, year). Available only when scraping with accounts |
| Remove bad accounts | ☑ | Automatic removal of accounts with incorrect login/password or requiring phone confirmation |