Skip to main content

SE::Google::KeywordPlanner::Ideas - scraper of keyword suggestions from Google Keyword Planner

Keyword Planner ideas

Overview of the scraper

SE::Google::KeywordPlanner::IdeasSE::Google::KeywordPlanner::Ideas – scraping keyword suggestions and hints from Google Keyword Planner. A variety of data is available for use: scraping lists of hints, assessing competition for keywords, collecting average monthly search volumes, minimum and maximum bids, searching for new keywords of similar themes. In the scraping settings, you can specify the language, region, location, and data sampling period. Batch mode is also available, which allows you to obtain data for 20 keywords in one request to the service, thereby significantly speeding up the scraping process.

Thanks to the multi-threaded operation of A-Parser, the speed of processing requests can reach several thousand requests per minute.

A-Parser functionality allows you to save the scraper settings of SE::Google::KeywordPlanner::Ideas for further use (presets), set a scraping schedule, and much more.

Saving results is possible in the form and structure that you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Collected data

  • Average monthly search volume for the target keyword
  • Lists of hints
  • Keyword suggestions
    • Average monthly search volume
    • Competition
    • Minimum and maximum bids
    • Trends for each obtained variant


  • Support for authentication by login-password or through substitution of cookies and headers
  • Batch mode is supported, more details in the section Queries
  • Support for multi-accounts (to select the required account, its ocid(uscid) must be specified)
  • Scraping data on search volumes for each keyword monthly for the specified period ($ideas.$i.trends). Data is presented in JSON, an example of their output in the result is shown in the screenshot below:
Spoiler: Screenshot

example of trends output in the result


[% FOREACH item IN ideas;
trends = tools.parseJSON(item.trends);

item.keyword _ "\n";
trends.format('$month: $volume\n');
END %]

Use cases

  • Scraping lists of hints
  • Assessing competition for keywords
  • Collecting average monthly search volumes, minimum and maximum bids
  • Searching for new keywords of similar themes


There are two options for configuring the scraper:

  • specify the email\password from the Keyword Planner account
  • log in through the browser and copy the required values

Be cautious with the number of threads. It is recommended to specify a small number of threads, while scraping without proxies is quite possible.

Authorization via email and password

You need to override the E-mail and Password options by specifying the data from your Keyword Planner account. A campaign must be created on the account.

Spoiler: (Solution) Login failed TypeError: Cannot read property '1' of null

In case of this error, you need to remove your Google account from the browser and log in again.

solution to authorization error

Browser authorization and header substitution in the scraper

It is necessary to log in to the browser via the link, create the first campaign if it has not been done before, take the following data and specify them in the scraper settings:

Cookies can be specified in two ways:

  • Specify all cookies in the All cookies option
  • Specify values from cookies for the options __Secure-3PSID, __Secure-3PSIDTS (__Secure-3PSIDTS should be specified if authuser on the account is equal to 0)

Other headers:

  • The value of the x-framework-xsrf-token header
  • The value of the ocid or uscid parameter from the URL
  • The value of the authuser parameter from the URL
Spoiler: How to find the necessary parameters

where to look for authorization parameters 1

where to look for authorization parameters 2


Depending on the value of the Query type parameter, queries can have different forms. Below are the possible options, examples are shown, and the features of the obtained results are described.


Queries should be in the form of keywords, one KS per line. Example of queries:

Windows 11
как вырастить дерево

Bulk (packet) mode is supported, which is activated by the Bulk (packet) mode option. In this mode, the scraper will send packets of 20 KS in a request to the service, which changes the logic of filling in the results:

  • $volume will be filled for each KS
  • $ideas and $suggests will be filled only for the first KS, but these arrays will contain all the results cumulatively for all KS used in this packet

Site + keyword

Queries should be in the form of a site and a keyword separated by a space. Example of queries: Network speed parser

Bulk (packet) mode is also supported, to use it you need to list keywords separated by commas, example: android,ios,прошивка google,ads,реклама,поиск сайтов в интернете
  • $volume for this type of queries in bulk mode is not parsed

Entire site

As queries, you need to specify domains, one per line. For example:
  • $volume for this type of queries is not parsed


As queries, you need to specify links, one per line. For example:
  • $volume for this type of queries is not parsed

Query substitutions

You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add some list of other words to each query, let's specify several main queries:

tower defense

In the query format, we will specify a macro for substituting additional words from the file keywords.txt, this method allows to increase the variability of queries many times over:

{subs:keywords} $query 

This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro's work.

For example, if the file keywords.txt will contain:


As a result, the substitution macro will turn 3 main queries into 6:

free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg

Output results examples

A-Parser supports flexible formatting of results thanks to the built-in Template Toolkit templating engine, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON

Default Output

Result format:


Example of result:

coca cola  
iphone 11 pro
iphone 11 pro max
winter season
iphone 11 price
apple iphone 11
iphone 11pro
11 pro max
iphone 11 pro price
iphone 11 max
iphone pro max
iphone 11 128gb
11 pro
iphone 11 pro max price
apple iphone 11 pro
apple iphone 11 pro max
new iphone 11
iphone 11 max pro
apple 11 pro
iphone 11 deals
iphone 11 pro max 256gb
diet coke
first day of winter
iphone 11 pro 256gb
coke zero
iphone pro 11
apple 11 pro max

Output in CSV Table

Result format:

[% FOREACH i IN ideas; 
tools.CSVline(i.keyword, i.volume, i.min_bid, i.max_bid);
END %]

File name:


Initial text:

Keyword,Volume,"Min bid","Max bid"


In the Result format, the Template Toolkit is used to output the $ideas array in a FOREACH loop.

In the file name of the results, you just need to change the file extension to csv.

To make the "Initial text" option available in the Task Editor, you need to activate "More options". In the "Initial text", we write the names of the columns separated by commas and make the second line empty.

Saving in SQL Format

Result format:

[% FOREACH ideas;
"INSERT INTO ideas VALUES('" _ keyword _ "', '" _ volume _ "')\n";
END %]

Example of result:

INSERT INTO ideas VALUES('духи', '50000')
INSERT INTO ideas VALUES('eyfel perfume', '5000')
INSERT INTO ideas VALUES('memo marfa', '5000')
INSERT INTO ideas VALUES('duxi', '5000')
INSERT INTO ideas VALUES('kenzo intense', '5000')
INSERT INTO ideas VALUES('climat lancome', '5000')
INSERT INTO ideas VALUES('v canto', '5000')
INSERT INTO ideas VALUES('majda bekkali', '5000')
INSERT INTO ideas VALUES('v canto ricina', '500')
INSERT INTO ideas VALUES('v canto stramonio', '5000')
INSERT INTO ideas VALUES('terenzi kirke', '500')
INSERT INTO ideas VALUES('duhi', '500')
INSERT INTO ideas VALUES('max mara le parfum', '500')
INSERT INTO ideas VALUES('stramonio v canto', '500')
INSERT INTO ideas VALUES('sheikh parfum', '500')
INSERT INTO ideas VALUES('jacques zolty', '500')
INSERT INTO ideas VALUES('aj arabia', '500')
INSERT INTO ideas VALUES('christian lacroix bazar', '500')
INSERT INTO ideas VALUES('juliette has a gun romantina', '500')
INSERT INTO ideas VALUES('vilhelm parfumerie mango skin', '500')
INSERT INTO ideas VALUES('v canto mirabile', '500')
INSERT INTO ideas VALUES('donna karan dkny be delicious', '500')
INSERT INTO ideas VALUES('arteolfatto', '500')
INSERT INTO ideas VALUES('aquawoman rochas', '500')
INSERT INTO ideas VALUES('angel and demon givenchy', '500')
INSERT INTO ideas VALUES('venenum kiss', '500')
INSERT INTO ideas VALUES('v canto mandragola', '500')
INSERT INTO ideas VALUES('angel demon givenchy', '500')
INSERT INTO ideas VALUES('hugo boss boss ma vie pour femme', '500')
INSERT INTO ideas VALUES('nina ricci mademoiselle ricci', '500')

Dump Results to JSON

General result format:

[% data = [];
FOREACH p1.ideas;
item = {};
item.keyword = keyword;
item.volume = volume;
END %]$data.json\n

Example of result:

[{"keyword":"духи","volume":"50000"},{"keyword":"eyfel perfume","volume":"5000"},{"keyword":"memo marfa","volume":"5000"},{"keyword":"duxi","volume":"5000"},{"keyword":"kenzo intense","volume":"5000"},{"keyword":"climat lancome","volume":"5000"},{"keyword":"v canto","volume":"5000"},{"keyword":"majda bekkali","volume":"5000"},{"keyword":"v canto ricina","volume":"500"},{"keyword":"v canto stramonio","volume":"5000"},{"keyword":"terenzi kirke","volume":"500"},{"keyword":"duhi","volume":"500"},{"keyword":"max mara le parfum","volume":"500"},{"keyword":"stramonio v canto","volume":"500"},{"keyword":"sheikh parfum","volume":"500"},{"keyword":"jacques zolty","volume":"500"},{"keyword":"aj arabia","volume":"500"},{"keyword":"christian lacroix bazar","volume":"500"},{"keyword":"juliette has a gun romantina","volume":"500"},{"keyword":"vilhelm parfumerie mango skin","volume":"500"},{"keyword":"v canto mirabile","volume":"500"},{"keyword":"donna karan dkny be delicious","volume":"500"},{"keyword":"arteolfatto","volume":"500"},{"keyword":"aquawoman rochas","volume":"500"},{"keyword":"angel and demon givenchy","volume":"500"},{"keyword":"venenum kiss","volume":"500"},{"keyword":"v canto mandragola","volume":"500"},{"keyword":"angel demon givenchy","volume":"500"},{"keyword":"hugo boss boss ma vie pour femme","volume":"500"},{"keyword":"nina ricci mademoiselle ricci","volume":"500"},{"keyword":"mmmm juliette has a gun","volume":"500"},{"keyword":"v canto lucrethia","volume":"500"},{"keyword":"mango skin vilhelm parfumerie","volume":"500"},{"keyword":"dalissime salvador dali","volume":"500"},{"keyword":"molecula 02","volume":"50000"},{"keyword":"lucia parfum","volume":"500"},{"keyword":"boadicea pure narcotic","volume":"500"},{"keyword":"terenzi andromeda","volume":"500"}]

More about outputting results in JSON is described in this article.

Possible Settings

ParameterDefault ValueDescription
All cookiesSpecify all cookies
Cookie "__Secure-3PSID"Cookie "__Secure-3PSID"
Cookie "__Secure-3PSIDTS"Cookie "__Secure-3PSIDTS"
Header "x-framework-xsrf-token"Header "x-framework-xsrf-token"
Url parameter "ocid"("uscid")Parameter "ocid"("uscid")
Url parameter "authuser"0Parameter "authuser"
E-mailE-mail for authorization in Keyword Planner
PasswordPassword for authorization in Keyword Planner
Recovery e-mailE-mail for account recovery
Browser headless (debug auth)Headless mode for the browser used for login-password authorization
Log Login Screenshot (debug auth)Creating a screenshot of the authorization page and displaying it in the task log
Date fromLast 12 monthsDate from
Date toLast 12 monthsDate to
Search networksGoogleSearch network
Location codeLocation (here you need to specify the location id, which can be taken from the first column of this table (copy))
Query typeKeywordQuery type
Exclude brand names in resultsBrand filter
Exclude adult ideasAdult content filter
Bulk (packet) modeEnable bulk mode