SE::Google::KeywordPlanner::Ideas - scraper of keyword suggestions from Google Keyword Planner

Keyword Planner ideas

Overview of the scraper

SE::Google::KeywordPlanner::Ideas – scraping keyword suggestions and hints from Google Keyword Planner. A variety of data is available for use: scraping lists of hints, assessing competition for keywords, collecting average monthly search volumes, minimum and maximum bids, searching for new keywords of similar themes. In the scraping settings, you can specify the language, region, location, and data sampling period. Batch mode is also available, which allows you to obtain data for 20 keywords in one request to the service, thereby significantly speeding up the scraping process.

Thanks to the multi-threaded operation of A-Parser, the speed of processing requests can reach several thousand requests per minute.

A-Parser functionality allows you to save the scraper settings of SE::Google::KeywordPlanner::Ideas for further use (presets), set a scraping schedule, and much more.

Saving results is possible in the form and structure that you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Go to DEMO Buy A-Parser Pro ($299)

Collected data

Average monthly search volume for the target keyword
Lists of hints
Keyword suggestions
- Average monthly search volume
- Competition
- Minimum and maximum bids
- Trends for each obtained variant

Capabilities

Support for authentication by login-password or through substitution of cookies and headers
Batch mode is supported, more details in the section Queries
Support for multi-accounts (to select the required account, its ocid(uscid) must be specified)
Scraping data on search volumes for each keyword monthly for the specified period ($ideas.$i.trends). Data is presented in JSON, an example of their output in the result is shown in the screenshot below:

Spoiler: Screenshot

example of trends output in the result

Template:

[% FOREACH item IN ideas;
    trends = tools.parseJSON(item.trends);
    
    item.keyword _ "\n";
    trends.format('$month: $volume\n');
    "----\n";
END %]

Use cases

Scraping lists of hints
Assessing competition for keywords
Collecting average monthly search volumes, minimum and maximum bids
Searching for new keywords of similar themes

Configuration

There are two options for configuring the scraper:

specify the email\password from the Keyword Planner account
log in through the browser and copy the required values

caution

Be cautious with the number of threads. It is recommended to specify a small number of threads, while scraping without proxies is quite possible.

Authorization via email and password

You need to override the E-mail and Password options by specifying the data from your Keyword Planner account. A campaign must be created on the account.

Spoiler: (Solution) Login failed TypeError: Cannot read property '1' of null

In case of this error, you need to remove your Google account from the browser and log in again.

solution to authorization error

Browser authorization and header substitution in the scraper

It is necessary to log in to the browser via the link https://ads.google.com/aw/keywordplanner/home, create the first campaign if it has not been done before, take the following data and specify them in the scraper settings:

Cookies can be specified in two ways:

Specify all cookies in the All cookies option
Specify values from cookies for the options __Secure-3PSID, __Secure-3PSIDTS (__Secure-3PSIDTS should be specified if authuser on the account is equal to 0)

Other headers:

The value of the x-framework-xsrf-token header
The value of the ocid or uscid parameter from the URL
The value of the authuser parameter from the URL

Spoiler: How to find the necessary parameters

where to look for authorization parameters 1

where to look for authorization parameters 2

Queries

Depending on the value of the Query type parameter, queries can have different forms. Below are the possible options, examples are shown, and the features of the obtained results are described.

`Keyword`

Queries should be in the form of keywords, one KS per line. Example of queries:

test
парсер
Windows 11
как вырастить дерево

Bulk (packet) mode is supported, which is activated by the Bulk (packet) mode option. In this mode, the scraper will send packets of 20 KS in a request to the service, which changes the logic of filling in the results:

$volume will be filled for each KS
$ideas and $suggests will be filled only for the first KS, but these arrays will contain all the results cumulatively for all KS used in this packet

`Site + keyword`

Queries should be in the form of a site and a keyword separated by a space. Example of queries:

speedtest.com Network speed
a-parser.com parser

Bulk (packet) mode is also supported, to use it you need to list keywords separated by commas, example:

4pda.to android,ios,прошивка
google.com google,ads,реклама,поиск сайтов в интернете

$volume for this type of queries in bulk mode is not parsed

`Entire site`

As queries, you need to specify domains, one per line. For example:

apple.com
microsoft.com

$volume for this type of queries is not parsed

`URL`

As queries, you need to specify links, one per line. For example:

https://a-parser.com/docs/parsers/se-google-keywordplanner
https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/JavaScript_basics

$volume for this type of queries is not parsed

Query substitutions

You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add some list of other words to each query, let's specify several main queries:

fantasy
tower defense
rpg

In the query format, we will specify a macro for substituting additional words from the file keywords.txt, this method allows to increase the variability of queries many times over:

{subs:keywords} $query

This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro's work.

For example, if the file keywords.txt will contain:

free
online

As a result, the substitution macro will turn 3 main queries into 6:

free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg

Output results examples

A-Parser supports flexible formatting of results thanks to the built-in Template Toolkit templating engine, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON

Default Output

Result format:

$ideas.format('$keyword\n')

Example of result:

coca cola  
iphone 11 pro  
winter  
iphone 11 pro max  
winter season  
iphone11  
iphone 11 price  
apple iphone 11  
iphone 11pro  
coke  
11 pro max  
iphone 11 pro price  
iphone 11 max
iphone pro max  
iphone 11 128gb  
11 pro  
iphone 11 pro max price  
apple iphone 11 pro  
apple iphone 11 pro max  
new iphone 11  
iphone 11 max pro  
apple 11 pro  
iphone 11 deals  
iphone 11 pro max 256gb  
diet coke  
first day of winter  
iphone 11 pro 256gb  
coke zero  
iphone pro 11  
apple 11 pro max

Output in CSV Table

Result format:

[% FOREACH i IN ideas; 
    tools.CSVline(i.keyword, i.volume, i.min_bid, i.max_bid);
END %]

File name:

$datefile.format().csv

Initial text:

Keyword,Volume,"Min bid","Max bid"

tip

In the Result format, the Template Toolkit is used to output the $ideas array in a FOREACH loop.

In the file name of the results, you just need to change the file extension to csv.

To make the "Initial text" option available in the Task Editor, you need to activate "More options". In the "Initial text", we write the names of the columns separated by commas and make the second line empty.

Saving in SQL Format

Result format:

[% FOREACH ideas;
    "INSERT INTO ideas VALUES('" _ keyword _ "', '" _ volume _ "')\n";
END %]

Example of result:

INSERT INTO ideas VALUES('духи', '50000')
INSERT INTO ideas VALUES('eyfel perfume', '5000')
INSERT INTO ideas VALUES('memo marfa', '5000')
INSERT INTO ideas VALUES('duxi', '5000')
INSERT INTO ideas VALUES('kenzo intense', '5000')
INSERT INTO ideas VALUES('climat lancome', '5000')
INSERT INTO ideas VALUES('v canto', '5000')
INSERT INTO ideas VALUES('majda bekkali', '5000')
INSERT INTO ideas VALUES('v canto ricina', '500')
INSERT INTO ideas VALUES('v canto stramonio', '5000')
INSERT INTO ideas VALUES('terenzi kirke', '500')
INSERT INTO ideas VALUES('duhi', '500')
INSERT INTO ideas VALUES('max mara le parfum', '500')
INSERT INTO ideas VALUES('stramonio v canto', '500')
INSERT INTO ideas VALUES('sheikh parfum', '500')
INSERT INTO ideas VALUES('jacques zolty', '500')
INSERT INTO ideas VALUES('aj arabia', '500')
INSERT INTO ideas VALUES('christian lacroix bazar', '500')
INSERT INTO ideas VALUES('juliette has a gun romantina', '500')
INSERT INTO ideas VALUES('vilhelm parfumerie mango skin', '500')
INSERT INTO ideas VALUES('v canto mirabile', '500')
INSERT INTO ideas VALUES('donna karan dkny be delicious', '500')
INSERT INTO ideas VALUES('arteolfatto', '500')
INSERT INTO ideas VALUES('aquawoman rochas', '500')
INSERT INTO ideas VALUES('angel and demon givenchy', '500')
INSERT INTO ideas VALUES('venenum kiss', '500')
INSERT INTO ideas VALUES('v canto mandragola', '500')
INSERT INTO ideas VALUES('angel demon givenchy', '500')
INSERT INTO ideas VALUES('hugo boss boss ma vie pour femme', '500')
INSERT INTO ideas VALUES('nina ricci mademoiselle ricci', '500')

Dump Results to JSON

General result format:

[% data = [];
FOREACH p1.ideas;
    item = {};
    item.keyword = keyword;
    item.volume = volume;
    data.push(item);
END %]$data.json\n

Example of result:

[{"keyword":"духи","volume":"50000"},{"keyword":"eyfel perfume","volume":"5000"},{"keyword":"memo marfa","volume":"5000"},{"keyword":"duxi","volume":"5000"},{"keyword":"kenzo intense","volume":"5000"},{"keyword":"climat lancome","volume":"5000"},{"keyword":"v canto","volume":"5000"},{"keyword":"majda bekkali","volume":"5000"},{"keyword":"v canto ricina","volume":"500"},{"keyword":"v canto stramonio","volume":"5000"},{"keyword":"terenzi kirke","volume":"500"},{"keyword":"duhi","volume":"500"},{"keyword":"max mara le parfum","volume":"500"},{"keyword":"stramonio v canto","volume":"500"},{"keyword":"sheikh parfum","volume":"500"},{"keyword":"jacques zolty","volume":"500"},{"keyword":"aj arabia","volume":"500"},{"keyword":"christian lacroix bazar","volume":"500"},{"keyword":"juliette has a gun romantina","volume":"500"},{"keyword":"vilhelm parfumerie mango skin","volume":"500"},{"keyword":"v canto mirabile","volume":"500"},{"keyword":"donna karan dkny be delicious","volume":"500"},{"keyword":"arteolfatto","volume":"500"},{"keyword":"aquawoman rochas","volume":"500"},{"keyword":"angel and demon givenchy","volume":"500"},{"keyword":"venenum kiss","volume":"500"},{"keyword":"v canto mandragola","volume":"500"},{"keyword":"angel demon givenchy","volume":"500"},{"keyword":"hugo boss boss ma vie pour femme","volume":"500"},{"keyword":"nina ricci mademoiselle ricci","volume":"500"},{"keyword":"mmmm juliette has a gun","volume":"500"},{"keyword":"v canto lucrethia","volume":"500"},{"keyword":"mango skin vilhelm parfumerie","volume":"500"},{"keyword":"dalissime salvador dali","volume":"500"},{"keyword":"molecula 02","volume":"50000"},{"keyword":"lucia parfum","volume":"500"},{"keyword":"boadicea pure narcotic","volume":"500"},{"keyword":"terenzi andromeda","volume":"500"}]

tip

More about outputting results in JSON is described in this article.

Possible Settings

note

General settings for all scrapers

Parameter	Default Value	Description
All cookies		Specify all cookies
Cookie "__Secure-3PSID"		Cookie "__Secure-3PSID"
Cookie "__Secure-3PSIDTS"		Cookie "__Secure-3PSIDTS"
Header "x-framework-xsrf-token"		Header "x-framework-xsrf-token"
Url parameter "ocid"("uscid")		Parameter "ocid"("uscid")
Url parameter "authuser"	`0`	Parameter "authuser"
E-mail		E-mail for authorization in Keyword Planner
Password		Password for authorization in Keyword Planner
Recovery e-mail		E-mail for account recovery
Browser headless (debug auth)	`☑`	Headless mode for the browser used for login-password authorization
Log Login Screenshot (debug auth)	`☐`	Creating a screenshot of the authorization page and displaying it in the task log
Date from	`Last 12 months`	Date from
Date to	`Last 12 months`	Date to
Language	`English`	Language
Search networks	`Google`	Search network
Currency	`USD`	Currency
Location code		Location (here you need to specify the location id, which can be taken from the first column of this table (copy))
Query type	`Keyword`	Query type
Exclude brand names in results	`☐`	Brand filter
Exclude adult ideas	`☑`	Adult content filter
Bulk (packet) mode	`☐`	Enable bulk mode

Overview of the scraper​

Collected data​

Capabilities​

Use cases​

Configuration​

Authorization via email and password​

Browser authorization and header substitution in the scraper​

Queries​

Keyword​

Site + keyword​

Entire site​

URL​

Query substitutions​

Output results examples​

Default Output​

Output in CSV Table​

Saving in SQL Format​

Dump Results to JSON​

Possible Settings​

Overview of the scraper

Collected data

Capabilities

Use cases

Configuration

Authorization via email and password

Browser authorization and header substitution in the scraper

Queries

`Keyword`

`Site + keyword`

`Entire site`

`URL`

Query substitutions

Output results examples

Default Output

Output in CSV Table

Saving in SQL Format

Dump Results to JSON

Possible Settings