SE::Google::KeywordPlanner::SearchVolume - evaluating search volume and other metrics for keywords from Google Keyword Planner
Overview of the scraper
SE::Google::KeywordPlanner::SearchVolume – scraper for estimating the search volume of keywords from Google Keyword Planner. A variety of retrieved data is available for use: average monthly queries, quarter-over-quarter and year-over-year volume changes, minimum and maximum bids, and competition level. In the parsing settings, you can specify the language, location, advertising network, and data selection period. Batch mode is also available, allowing one request to retrieve data for up to 10000 keywords, significantly speeding up the scraping process.Thanks to the multithreaded operation of A-Parser combined with batch mode, the query processing speed can reach tens of thousands of requests per minute.
A-Parser's functionality allows you to save the parsing settings of the SE::Google::KeywordPlanner::SearchVolume scraper for further use (presets), set a parsing schedule, and much more.
Results can be saved in the format and structure you require, thanks to the built-in powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.
Collected data
- Average monthly search volume for the searched keyword
- Quarter-over-quarter change
- Year-over-year change
- Competition level
- Minimum and maximum bids
- Search volume trends for the selected period
- Month and year
- Total number of queries
- Number of queries from mobile devices only (not available in batch mode)
Capabilities
- Support for authorization via login/password or by substituting cookies and headers
- Determining the precision of $volume - exact/rounded value
- Batch mode is supported, enabled by default
- Support for multi-accounts (the
ocid(uscid)must be specified to select the correct account)
Use Cases
- Gathering average monthly queries, minimum and maximum bids, and competition assessment
- Trend analysis
Setup
There are two options for configuring the scraper:
- provide the email/password for the Keyword Planner account
- authorize in the browser and copy the necessary values
Be cautious with the number of threads. It is recommended to specify a small number of threads, and parsing without proxies is possible.
Authorization via email and password
You need to override the options E-mail and Password, by specifying the data from your Keyword Planner account. A campaign must be created on the account.
Spoiler: (Solution) Login failed TypeError: Cannot read property '1' of null
If this error occurs, you need to delete your Google account from the browser and log in again.
Browser authorization and header substitution in the scraper
You must authorize in the browser using the link https://ads.google.com/aw/keywordplanner/home, create the first campaign if none existed before, fetch the following data, and specify it in the scraper settings:
Cookies can be specified in two ways:
- Specify all cookies in the All cookies option
- Specify cookie values for options __Secure-3PSID, __Secure-3PSIDTS (__Secure-3PSIDTS must be specified if authuser on the account is 0)
Other headers:
- Value of the x-framework-xsrf-token header
- Value of the ocid or uscid parameter from the URL
- Value of the authuser parameter from the URL
Spoiler: How to find the necessary parameters
Queries
Keywords should be specified as queries, one keyword per line. Example queries:
coca-cola
parsing
peace
starlink
test
The batch mode, which is enabled by the option Bulk (packet) mode. is supported. In this mode, the scraper will send packages of 10000 keywords in a single request to the service. Data on the number of queries from mobile devices is not collected in this mode ($trends.$i.mobile). Batch mode is enabled by default.
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files; for example, if we want to add some list of other words to each query, we specify a few main queries:
fantasy
tower defense
rpg
In the query format, we specify the macro to substitute additional words from the file keywords.txt, this method can multiply the query variations:
{subs:keywords} $query
This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro operation.
For example, if the file keywords.txt contains:
free
online
As a result, the substitution macro will turn 3 main queries into 6:
free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg
Output results examples
A-Parser supports flexible results formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Default output
Result format:
$query: $volume\n
Example result showing the keyword and its average monthly search volume:
coca-cola: 3350000
parsing: 165000
peace: 673000
starlink: 1830000
test: 7480000
Output to CSV table
General result format:
[% tools.CSVline(query, p1.volume, p1.min_bid, p1.max_bid) %]
File name:
$datefile.format().csv
Initial text:
Keyword,Volume,"Min bid","Max bid"
In the General result format, the Template Toolkit templating engine and the tools.CSVline tool are used for data output.
In the results file name, you simply need to change the file extension to csv.
For the "Initial text" option to be available in the Task Editor, you need to activate "More options". In "Initial text", write the column names separated by commas, and the second line should be empty.
Saving in SQL format
Result format:
[% "INSERT INTO volumes VALUES('" _ query _ "', '" _ volume _ "')\n" %]
Example result:
INSERT INTO volumes VALUES('perfumes', '50000')
INSERT INTO volumes VALUES('eyfel perfume', '5000')
INSERT INTO volumes VALUES('memo marfa', '5000')
Dump results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.keyword = query;
obj.volume = p1.volume;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example result:
[{"keyword":"perfumes","volume":"50000"},
{"keyword":"eyfel perfume","volume":"5000"},
{"keyword":"memo marfa","volume":"5000"}]
More details on outputting results to JSON are described in this article.
Possible settings
| Parameter | Default value | Description |
|---|---|---|
| All cookies | Specify all cookies | |
| Cookie "__Secure-3PSID" | Cookie "__Secure-3PSID" | |
| Cookie "__Secure-3PSIDTS" | Cookie "__Secure-3PSIDTS" | |
| Header "x-framework-xsrf-token" | Header "x-framework-xsrf-token" | |
| Url parameter "ocid"("uscid") | Parameter "ocid"("uscid") | |
| Url parameter "authuser" | 0 | Parameter "authuser" |
| E-mail for Keyword Planner authorization | ||
| Password | Password for Keyword Planner authorization | |
| Recovery e-mail | Recovery e-mail | |
| Browser headless (debug auth) | ☑ | Headless mode for the browser used for login/password authorization |
| Log Login Screenshot (debug auth) | ☐ | Create a screenshot of the authorization page and output it to the task log |
| Date from | Last 12 months | Date from |
| Date to | Last 12 months | Date to |
| Language | English | Language |
| Search networks | Google | Search network |
| Location code | Location (here you need to specify the location ID, which can be taken from the first column of this table (copy)) | |
| Delete created plan | ☑ | Delete the created plan |
| Bulk (packet) mode | ☑ | Enable batch mode |