Skip to main content

SE::Yandex::WordCraft - WordCraft Scraper. Query Selection and Market Analysis

img

Overview of the scraper​

Yandex WordCraft (WordCraft) is a part of the Yandex Webmaster service that allows you to select target queries for promotion and advertising, taking into account popularity and competition. Hundreds of millions of unique queries are available for analysis. Demand, impressions, clicks, and positions are averaged over the month for the past year.

The Yandex WordCraft query scraper supports automatic query expansion; you can be sure to get the maximum number of results from the output.

A-Parser functionality allows you to save parsing settings for later use (presets), set a parsing scheduler, and much more. You can use automatic query expansion, substitution of sub-queries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results when scraping Yandex Wordstat.

Results can be saved in the form and structure you need, thanks to the built-in powerful templater Template Toolkit which allows you to apply additional logic to results and output data in various formats, including TXT, JSON, SQL and CSV.

Accounts​

For the SE::Yandex::WordCraftSE::Yandex::WordCraft scraper to work, Yandex accounts are required. Accounts can be registered using the SE::Yandex::RegisterSE::Yandex::Register scraper or simply add existing accounts to the files/SE-Yandex/accounts.txt accounts.txt file in a supported format.

Alternatively, you can enable account registration "on the fly".

To work using session authentication, the data string must be in the following format: [email protected];MAQT78Z31Rinx4H;{"answer":"qmfhsxdcrk","proxy":"185.104.120.45:3128","session_id":"3:1748440908.5.0.1748440867459:ZXBxpg:47e4.1.2:1|2191075974.41.2.2:41.3:1748440908|3:10308131.797655.5pfkoRZWgLJGntKTlcUhYdysNfk"}

Scraped data​

  • User requests (main):
    • Query, competitiveness, demand, clicks
  • User requests (additional):
    • Query, competitiveness, demand, clicks
  • Popular sites in Yandex search (sites):
    • Site, impression frequency, average position
  • Popular sites in Yandex search (pages):
    • URL, impression frequency, average position, number of queries, title

Use cases​

  • Keyword competitiveness assessment (demand, clicks)
  • Searching for new keywords with similar themes
  • Collecting large databases of keywords of different themes
  • Any other options involving scraping Yandex Webmaster WordCraft in one form or another

Queries​

Queries must specify keywords, exactly as if they were entered directly into the WordCraft search field, for example:

vacuum cleaner
seo

Result output options​

A-Parser supports flexible result formatting thanks to the built-in templater Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured form, such as CSV or JSON

Default output​

Example of outputting a list of user queries (main). Result format:

User queries general: $uqg.format('$query,$compet,$demand,$clicks\n')

Result is a list:

vacuum cleaner,HIGH,33190,9248
vacuum cleaner,HIGH,31204,8768
vacuum cleaner,HIGH,14097,6082
...  
vacuum cleaner,HIGH,791,656
note

To output results using Common Result Format you need to specify the scraper number in the task - $p1. (parser 1) or p1. if the Template Toolkit templater. is used. For example: 

User queries general: $p1.uqg.format('$query,$compet,$demand,$clicks\n')

See Edit Results Format for what Common Result Format is

Output to CSV table​

Example of outputting a list of user queries (main) to a CSV table. Result format:

[%  
    FOREACH uqg; 
        tools.CSVline(query,compet,demand,clicks); 
    END;
%]

Example of outputting a list of user queries (additional) to a CSV table. Result format:

[%  
    FOREACH uqa; 
        tools.CSVline(query,compet,demand,clicks); 
    END;
%]

Example of outputting a list of popular sites in Yandex search (sites) to a CSV table. Result format:

[%  
    FOREACH rs; 
        tools.CSVline(site,popul,position); 
    END;
%]

Example of outputting a list of popular sites in Yandex search (pages) to a CSV table. Result format:

[%  
    FOREACH rp; 
        tools.CSVline(url,popul,position,queries,title); 
    END
%]
tip

See also: Result Filters

Options​

  • Parse queries for Rivals pages - when this option is enabled, popular queries are collected for each of the links in Rivals pages. The collected data is written as a JSON object to the variable $rp.$i.queries_json. Enabling this option significantly increases parsing time, as the scraper makes many additional sub-requests.

Possible settings​

note

Common settings for all scrapers

ParameterDefault ValueDescription
AntiGate presetdefaultYou need to pre-configure the Util::AntiGateUtil::AntiGate scraper - specify your access key and other parameters, then select the created preset here
AntiGate preset for LogindefaultAntiGate preset for login. You need to pre-configure the Util::AntiGateUtil::AntiGate scraper with parameters, then select the created preset here
AccountsOnly from "accounts.txt"Select account method: Always auto register - always automatically register accounts on the fly, requires selecting a configured preset in the SE::Yandex::Register preset parameter. Auto register if no more in "accounts.txt" - first use existing accounts from accounts.txt, and if they run out - use automatic registration on the fly, for which you need to select a configured preset in the SE::Yandex::Register preset parameter. Only from "accounts.txt" - use only existing accounts from accounts.txt, and if they run out - wait for the specified time (Wait new accounts in "accounts.txt" parameter) for new ones to appear. Only by session_id from "accounts.txt" - authentication by cookies.
Wait new accounts in "accounts.txt"0Waiting time for new accounts to appear in accounts.txt
Remove bad accountsAlways, except wrong login/passwordAutomatic deletion of "bad" accounts: Always - always delete. Always, except wrong login/password - always delete, except when Yandex reported incorrect login/password. The thing is, Yandex can send such a message upon IP ban for an absolutely working account, so optionally you can leave such accounts for reuse. Never - never delete. Regardless of the chosen option, accounts are not deleted in case of proxy/browser errors
SE::Yandex::Register presetdefaultSelect preset settings for SE::Yandex::RegisterSE::Yandex::Register
Use sessions☑Use sessions
Do not reset session if authorization passed☑Do not reset session on errors if the scraper has already authenticated
SE::Yandex::Register presetdefaultSelect preset settings for SE::Yandex::RegisterSE::Yandex::Register