SE::Google::KeywordPlanner::SearchVolume - estimation of the number of queries and other key word indicators from Google Keyword Planner
Overview of the scraper
SE::Google::KeywordPlanner::SearchVolume – a scraper for estimating the search volume of keywords with Google Keyword Planner. A variety of data is available for use: average number of queries per month, changes in volume over the quarter and year, minimum and maximum bids, competition level. In the scraping settings, you can specify the language, location, advertising network, and data sampling period. Batch mode is also available, which allows you to obtain data for up to 10,000 keywords in a single request to the service, thereby significantly speeding up the scraping process.Thanks to the multi-threaded operation of A-Parser in combination with batch mode, the speed of processing requests can reach tens of thousands of queries per minute.
A-Parser functionality allows you to save the scraping settings of the SE::Google::KeywordPlanner::SearchVolume scraper for future use (presets), set a scraping schedule, and much more.
Saving results is possible in the form and structure that you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
Collected data
- Average number of queries per month for the target keyword
- Change over the quarter
- Change over the year
- Competition level
- Minimum and maximum bids
- Search volume trends for the selected period
- Month and year
- Total number of queries
- Number of queries from mobile devices only (not available in batch mode)
Capabilities
- Support for authentication via login-password or through substitution of cookies and headers
- Batch mode supported, enabled by default
- Support for multi-accounts (to select the desired account, its
ocid(uscid)must be specified)
Use cases
- Collecting average number of queries per month, minimum and maximum bids, competition assessment
- Trend analysis
Configuration
There are two options for configuring the scraper:
- specify the email/password of the Keyword Planner account
- authenticate in the browser and copy the required values
Be cautious with the number of threads. It is recommended to specify a small number of threads, while parsing without proxies is quite possible.
Authentication via email and password
You need to override the E-mail and Password options by specifying the details of your Keyword Planner account. A campaign must be created on the account.
Spoiler: (Solution) Login failed TypeError: Cannot read property '1' of null
If you encounter this error, you need to remove your Google account from the browser and log in again.
Browser authentication and substitution of headers in the scraper
You need to log in to the browser using the link https://ads.google.com/aw/keywordplanner/home, create the first campaign if you haven't done so before, take the following data and specify them in the scraper settings:
Cookies can be specified in two ways:
- Specify all cookies in the All cookies option
- Specify the values from the cookie for the options __Secure-3PSID, __Secure-3PSIDTS (__Secure-3PSIDTS should be specified if authuser on the account is equal to 0)
Other headers:
- The value of the x-framework-xsrf-token header
- The value of the ocid or uscid parameter from the URL
- The value of the authuser parameter from the URL
Spoiler: How to find the necessary parameters
Queries
As queries, you need to specify keywords, one KW per line. Example of queries:
coca-cola
parsing
peace
starlink
test
Bulk (packet) mode is supported, which is activated by the Bulk (packet) mode option. In this mode, the scraper will send packets of 10000 KWs in a request to the service. In this mode, data on the number of queries from mobile devices ($trends.$i.mobile) are not collected. Bulk mode is enabled by default.
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add some list of other words to each query, let's specify several main queries:
fantasy
tower defense
rpg
In the query format, we will specify a macro for substituting additional words from the file keywords.txt, this method allows to increase the variability of queries many times over:
{subs:keywords} $query
This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro's work.
For example, if the file keywords.txt will contain:
free
online
As a result, the substitution macro will turn 3 main queries into 6:
free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg
Output Results Examples
A-Parser supports flexible formatting of results thanks to the built-in Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Default Output
Result format:
$query: $volume\n
An example of the result with the output of the keyword and its average number of queries per month:
coca-cola: 3350000
parsing: 165000
peace: 673000
starlink: 1830000
test: 7480000
Output in CSV Table
General result format:
[% tools.CSVline(query, p1.volume, p1.min_bid, p1.max_bid) %]
File name:
$datefile.format().csv
Starting text:
Keyword,Volume,"Min bid","Max bid"
In the General result format, the Template Toolkit and the tool tools.CSVline are used to output data.
In the file name of the results, you just need to change the file extension to csv.
To make the "Starting text" option available in the Task Editor, you need to activate "More options". In the "Starting text", we write the names of the columns separated by commas and make the second line empty.
Saving in SQL Format
Result format:
[% "INSERT INTO volumes VALUES('" _ query _ "', '" _ volume _ "')\n" %]
Example of the result:
INSERT INTO volumes VALUES('духи', '50000')
INSERT INTO volumes VALUES('eyfel perfume', '5000')
INSERT INTO volumes VALUES('memo marfa', '5000')
Dump Results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.keyword = query;
obj.volume = p1.volume;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example of the result:
[{"keyword":"духи","volume":"50000"},
{"keyword":"eyfel perfume","volume":"5000"},
{"keyword":"memo marfa","volume":"5000"}]
More details about outputting results in JSON are described in this article.
Possible Settings
| Parameter | Default Value | Description |
|---|---|---|
| All cookies | Specifying all cookies | |
| Cookie "__Secure-3PSID" | Cookie "__Secure-3PSID" | |
| Cookie "__Secure-3PSIDTS" | Cookie "__Secure-3PSIDTS" | |
| Header "x-framework-xsrf-token" | Header "x-framework-xsrf-token" | |
| Url parameter "ocid"("uscid") | Url parameter "ocid"("uscid") | |
| Url parameter "authuser" | 0 | Url parameter "authuser" |
| E-mail for authorization in Keyword Planner | ||
| Password | Password for authorization in Keyword Planner | |
| Recovery e-mail | E-mail for account recovery | |
| Browser headless (debug auth) | ☑ | Headless mode for the browser used for login-password authorization |
| Log Login Screenshot (debug auth) | ☐ | Taking a screenshot of the login page and logging it in the task log |
| Date from | Last 12 months | Date from |
| Date to | Last 12 months | Date to |
| Language | English | Language |
| Search networks | Google | Search network |
| Location code | Location (here you need to specify the location id, which can be taken from the first column of this table (copy)) | |
| Delete created plan | ☑ | Delete the created plan |
| Bulk (packet) mode | ☑ | Enabling bulk (packet) mode |