Skip to main content

SE::Yandex::WordStat::ByDate - Yandex WordStat impressions scraper by date

Yandex WordStat by date scraper overviewโ€‹

img

Wordstat is a Yandex service designed to evaluate user interest in various topics and select keywords for SEO optimization and contextual advertising. In addition, with Wordstat Yandex, you can evaluate the seasonality and geographical dependence of search queries.

The Yandex WordStat by date scraper supports automatic duplication of queries, so you can be sure that you will get the maximum number of results from the output. Also, A-Parser can automatically navigate related queries to the specified depth.

The A-Parser functionality allows you to save parsing settings for further use (presets), set a parsing schedule, and much more. You can use automatic duplication of queries, substitution of sub-queries from files, enumeration of alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the form and structure that you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Use cases for the Wordstat by date scraperโ€‹

Wordstat bydate automation

Used to store up-to-date information in the database and generate a csv with fresh data each time it is launched

Accountsโ€‹

To work with the SE::Yandex::WordStat::ByDateSE::Yandex::WordStat::ByDate scraper, Yandex accounts are required. Accounts can be registered using the SE::Yandex::RegisterSE::Yandex::Register scraper or simply add existing accounts to the files/SE-Yandex/accounts.txt file in the supported format.

Or you can enable registration of accounts "on the fly".

Collected dataโ€‹

  • Keyword statistics by month or week
  • Date
  • Absolute value
  • Relative value

Data collected by the SE::Yandex::WordStat::ByDate scraper

Capabilitiesโ€‹

  • Supports selection of search region (with subgroups)
  • Ability to select multiple regions for evaluation at once
  • Supports automatic Smart captcha bypass and the ability to bypass graphical captcha using the AntiCaptcha service or any other that supports their API
  • Choice of device type
  • Ability to choose authentication method
  • Ability to register accounts "on the fly"
  • Supports working with advanced account format and can answer the secret question (if the answer is in info). And also uses saved proxy for authorization (if it is in info).

Use casesโ€‹

  • Estimation of traffic volume by keyword
  • Identification of seasonal keywords

Queriesโ€‹

  • As queries, you need to specify keywords, just like if you entered them directly into the Wordstat search form, for example:

test

Resultsโ€‹

  • As a result, keyword statistics for the month and week are displayed:
Monthly:  
2011-09-30 3010832 0.0008903808
2011-10-31 681432 0.0001825883
2011-11-30 628532 0.0001575008
2011-12-31 629072 0.0001495699
2012-01-31 561206 0.0001300651
2012-02-29 572039 0.0001290000
2012-03-31 614897 0.0001225754
2012-04-30 520433 0.0001185340
2012-05-31 521967 0.0001235327
2012-06-30 502568 0.0001299958
...
Weekly:
2012-09-16 118715 0.0001222877
2012-09-23 120799 0.0001211773
2012-09-30 137809 0.0001365837
2012-10-07 133929 0.0001313643
2012-10-14 140373 0.0001293922
2012-10-21 136014 0.0001242209
2012-10-28 148350 0.0001293328
2012-11-04 139556 0.0001232566
2012-11-11 154830 0.0001314057
2012-11-18 136458 0.0001147489
2012-11-25 149463 0.0001261401
2012-12-02 144724 0.0001197564
2012-12-09 149142 0.0001212195
2012-12-16 162864 0.0001298181

Result output optionsโ€‹

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in any form, as well as in a structured form, such as CSV or JSON.

Outputting results in JSONโ€‹

Result format:

[%  data = {}; 
data.views = [];

FOREACH i IN views;
item = {};
item.date = i.date;
item.relcount = i.relcount;
item.count = i.count;
data.views.push(item);
END;

data.json %]

Result example:

{
"views": [
{
"count": "9661734",
"date": "2012-03-31",
"relcount": "0.0019259985"
},
{
"count": "8567243",
"date": "2012-04-30",
"relcount": "0.0019512785"
},
{
"count": "9028986",
"date": "2012-05-31",
"relcount": "0.0021368683"
}
]
}

Outputting results in CSVโ€‹

Result format:

[% FOREACH i IN views;
tools.CSVline(query, i.count, i.date);
END %]

Result example:

"ั‚ะตัั‚",9661734,2012-03-31
"ั‚ะตัั‚",8567243,2012-04-30
"ั‚ะตัั‚",9028986,2012-05-31
"ั‚ะตัั‚",6082099,2012-06-30
"ั‚ะตัั‚",5531950,2012-07-31
"ั‚ะตัั‚",5214663,2012-08-31
"ั‚ะตัั‚",6603865,2012-09-30
"ั‚ะตัั‚",9127457,2012-10-31
"ั‚ะตัั‚",9238652,2012-11-30

Dumping results to SQLโ€‹

Result format:

[% FOREACH i IN views;
"INSERT INTO views VALUES('" _ query _ "', '"; i.count _ "', '"; i.relcount _ "', '"; i.date _ "')\n";
END %]

Result example:

INSERT INTO serp VALUES('ั‚ะตัั‚', '9661734', '0.0019259985', '2012-03-31')
INSERT INTO serp VALUES('ั‚ะตัั‚', '8567243', '0.0019512785', '2012-04-30')
INSERT INTO serp VALUES('ั‚ะตัั‚', '9028986', '0.0021368683', '2012-05-31')
INSERT INTO serp VALUES('ั‚ะตัั‚', '6082099', '0.0015732140', '2012-06-30')
INSERT INTO serp VALUES('ั‚ะตัั‚', '5531950', '0.0013160071', '2012-07-31')
INSERT INTO serp VALUES('ั‚ะตัั‚', '5214663', '0.0013327945', '2012-08-31')
INSERT INTO serp VALUES('ั‚ะตัั‚', '6603865', '0.0015936909', '2012-09-30')
INSERT INTO serp VALUES('ั‚ะตัั‚', '9127457', '0.0018740506', '2012-10-31')
INSERT INTO serp VALUES('ั‚ะตัั‚', '9238652', '0.0018308715', '2012-11-30')
tip

See also: Results filters

Possible settingsโ€‹

ParameterDefault valueDescription
PeriodMonthlyPeriod selection
RegionAllSearch region
AntiGate presetdefaultYou need to configure the scraper Util::AntiGateUtil::AntiGate in advance - specify your access key and other parameters, and then select the created preset here
AntiGate preset for LogindefaultAntiGate preset for login. You need to configure the scraper Util::AntiGateUtil::AntiGate with parameters, and then select the created preset here
TypeAllDevice type selection
AccountsOnly from "accounts.txt"Account operation method selection: Always auto register - always automatically register accounts "on the fly", you need to select the configured preset in the parameter SE::Yandex::Register preset. Auto register if no more in "accounts.txt" - first use existing accounts from accounts.txt, and if they run out - use automatic registration "on the fly", for which you need to select the configured preset in the parameter SE::Yandex::Register preset. Only from "accounts.txt" - use only existing accounts from accounts.txt, and if they run out - wait for the specified time (parameter Wait new accounts in "accounts.txt") for new ones to appear
Wait new accounts in "accounts.txt"0Waiting time for new accounts to appear in accounts.txt
Remove bad accountsAlways, except wrong login/passwordAutomatic removal of "bad" accounts: Always - always remove. Always, except wrong login/password - always remove, except in cases where Yandex reported that the login/password is incorrect. The fact is that Yandex can give such a message when banning the IP for a completely working account, so such accounts can optionally be left for reuse. Never - never remove. Regardless of the selected option, accounts are not deleted in case of proxy/browser errors
SE::Yandex::Register presetdefaultSelection of settings preset for SE::Yandex::RegisterSE::Yandex::Register
Authorization methodHTTPAuthorization method: HTTP - fast, not resource-intensive. Chrome - slow, resource-intensive, theoretically can extend the life of accounts
Chrome headlessโ˜‘If the option is enabled, the browser will not be displayed
Use sessionsโ˜‘Session usage
Do not reset session if authorization passedโ˜‘Do not reset the session in case of errors if the scraper has already been authorized