SE::Google::KeywordPlanner::Ideas - Scraper for keyword ideas from Google Keyword Planner

Overview of the scraper

SE::Google::KeywordPlanner::Ideas – Scraping keyword ideas and suggestions from Google Keyword Planner. Many types of data are available for use: scraping suggestion lists, assessing keyword competition, collecting average monthly number of requests, minimum and maximum bids, finding new keywords of a similar topic. In the scraping settings, you can specify the language, region, location, and data sampling period. Batch mode is also available, allowing you to get data for up to 20 keywords at once per service request, significantly speeding up scraping.

Thanks to A-Parser's multithreading, the request processing speed can reach several thousand requests per minute.

A-Parser's functionality allows you to save the scraping settings for the SE::Google::KeywordPlanner::Ideas scraper for later use (presets), ), set a scraping schedule, and much more.

Saving results is possible in the format and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.

Go to DEMOBuy A-Parser Pro ($299)

Collected Data

• Average monthly search volume for the searched keyword
• Suggestion lists
• Keyword ideas
  • Average monthly search volume
  • Competition
  • Minimum and maximum bids
  • Trends for each resulting suggestion

Capabilities

• Supports authorization via login/password or by substituting cookies and headers
• Determines the precision of the $volume - exact/rounded value
• Batch mode is supported, more details in the section Queries
• Supports multi-accounts (to select the needed account, its ocid(uscid) must be specified)
• Scraping search volume data for each keyword on a monthly basis for the specified period ($ideas.$i.trends). The data is presented in JSON, an example of its output in the result is shown in the screenshot below:

Spoiler: Screenshot

Template:

[% FOREACH item IN ideas;
    trends = tools.parseJSON(item.trends);
    
    item.keyword _ "\n";
    trends.format('$month: $volume\n');
    "----\n";
END %]

Use Cases

• Scraping suggestion lists
• Assessing keyword competition
• Collection of average monthly number of requests, minimum and maximum bids
• Finding new keywords of similar topics

Setup

There are two options for setting up the scraper:

• Specify the email/password of the Keyword Planner account
• Authorize in the browser and copy the necessary values

caution

Be careful with the number of threads. It is recommended to specify a small number of threads, and scraping without proxies is possible.

Authorization via email and password

You need to override the options E-mail and Password, by specifying the data from your Keyword Planner account. A campaign must be created on the account.

Spoiler: (Solution) Login failed TypeError: Cannot read property '1' of null

If this error occurs, you need to remove your Google account from the browser and log in again.

Browser authorization and passing headers to the scraper

You need to authorize in the browser via the link https://ads.google.com/aw/keywordplanner/home, create the first campaign if none existed before, take the following data, and specify it in the scraper settings:

Cookies can be specified in two ways:

• Specify all cookies in the All cookies option
• Specify the values from the cookie for options __Secure-3PSID, __Secure-3PSIDTS (__Secure-3PSIDTS must be specified if authuser on the account is 0)

Other headers:

• Value of the x-framework-xsrf-token header
• Value of the ocid or uscid parameter from the URL
• Value of the authuser parameter from the URL

Spoiler: How to find the necessary parameters

Queries

Depending on the value of the parameter Query type queries may have a different format. The possible options, examples, and features of the resulting results are listed below.

Keyword

Queries should be in the form of keywords, one keyword per line. Example queries:

test
scraper
Windows 11
how to grow a tree

The batch mode, is supported, which is enabled by the option Bulk (packet) mode. In this mode, the scraper will send packages of 20 keywords in a request to the service, which changes the logic for filling the results:

• $volume will be populated for each keyword
• $ideas and $suggests will be filled only for the first keyword, but these arrays will contain all results summed up for all keywords used in this package

Site + keyword

Queries must be in the format of a site followed by a space and a keyword. Example queries:

speedtest.com Network speed
a-parser.com parser

The batch mode, is also supported; to use it, you need to list the keywords separated by commas, example:

4pda.to android,ios,firmware
google.com google,ads,advertising,website search on the internet

• $volume for this type of query in batch mode is not scraped

Entire site

Domains should be specified as queries, one per line. For example:

apple.com
microsoft.com

• $volume for this type of query is not scraped

URL

Links should be specified as queries, one per line. For example:

https://a-parser.com/docs/parsers/se-google-keywordplanner
https://developer.mozilla.org/en-US/docs/Learn/Getting_started_with_the_web/JavaScript_basics

• $volume for this type of query is not scraped

Query substitutions

You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add some list of other words to each query, let's specify a few main queries:

fantasy
tower defense
rpg

In the query format, we will specify the substitution macro for additional words from the file keywords.txt, this method allows multiplying the query variability:

{subs:keywords} $query 

This macro will create as many additional queries as there are in the file for each initial search query, which will result in a total of [number of initial queries] x [number of queries in the Keywords file] = [ total number of queries] as a result of the macro operation.

For example, if the file keywords.txt contains:

free
online

As a result, the substitution macro will turn 3 main queries into 6:

free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg

Output Results Examples

A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in structured formats, such as CSV or JSON

Default Output

Result format:

$ideas.format('$keyword\n')

Result example:

coca cola  
iphone 11 pro  
winter  
iphone 11 pro max  
winter season  
iphone11  
iphone 11 price  
apple iphone 11  
iphone 11pro  
coke  
11 pro max  
iphone 11 pro price  
iphone 11 max
iphone pro max  
iphone 11 128gb  
11 pro  
iphone 11 pro max price  
apple iphone 11 pro  
apple iphone 11 pro max  
new iphone 11  
iphone 11 max pro  
apple 11 pro  
iphone 11 deals  
iphone 11 pro max 256gb  
diet coke  
first day of winter  
iphone 11 pro 256gb  
coke zero  
iphone pro 11  
apple 11 pro max

Output to CSV Table

Result format:

[% FOREACH i IN ideas; 
    tools.CSVline(i.keyword, i.volume, i.min_bid, i.max_bid);
END %]

File name:

$datefile.format().csv

Initial text:

Keyword,Volume,"Min bid","Max bid"

tip

The Template Toolkit templating engine is used in the Results Format to output the $ideas array in a FOREACH loop.

In the results file name, you just need to change the file extension to csv.

To make the "Initial text" option available in the Job Editor, , you need to activate "More options". In "Initial text", we write the column names separated by commas and leave the second line empty.

Saving in SQL Format

Result format:

[% FOREACH ideas;
    "INSERT INTO ideas VALUES('" _ keyword _ "', '" _ volume _ "')\n";
END %]

Result example:

INSERT INTO ideas VALUES('perfume', '50000')
INSERT INTO ideas VALUES('eyfel perfume', '5000')
INSERT INTO ideas VALUES('memo marfa', '5000')
INSERT INTO ideas VALUES('duxi', '5000')
INSERT INTO ideas VALUES('kenzo intense', '5000')
INSERT INTO ideas VALUES('climat lancome', '5000')
INSERT INTO ideas VALUES('v canto', '5000')
INSERT INTO ideas VALUES('majda bekkali', '5000')
INSERT INTO ideas VALUES('v canto ricina', '500')
INSERT INTO ideas VALUES('v canto stramonio', '5000')
INSERT INTO ideas VALUES('terenzi kirke', '500')
INSERT INTO ideas VALUES('duhi', '500')
INSERT INTO ideas VALUES('max mara le parfum', '500')
INSERT INTO ideas VALUES('stramonio v canto', '500')
INSERT INTO ideas VALUES('sheikh parfum', '500')
INSERT INTO ideas VALUES('jacques zolty', '500')
INSERT INTO ideas VALUES('aj arabia', '500')
INSERT INTO ideas VALUES('christian lacroix bazar', '500')
INSERT INTO ideas VALUES('juliette has a gun romantina', '500')
INSERT INTO ideas VALUES('vilhelm parfumerie mango skin', '500')
INSERT INTO ideas VALUES('v canto mirabile', '500')
INSERT INTO ideas VALUES('donna karan dkny be delicious', '500')
INSERT INTO ideas VALUES('arteolfatto', '500')
INSERT INTO ideas VALUES('aquawoman rochas', '500')
INSERT INTO ideas VALUES('angel and demon givenchy', '500')
INSERT INTO ideas VALUES('venenum kiss', '500')
INSERT INTO ideas VALUES('v canto mandragola', '500')
INSERT INTO ideas VALUES('angel demon givenchy', '500')
INSERT INTO ideas VALUES('hugo boss boss ma vie pour femme', '500')
INSERT INTO ideas VALUES('nina ricci mademoiselle ricci', '500')

Dump Results to JSON

General result format:

[% data = [];
FOREACH p1.ideas;
    item = {};
    item.keyword = keyword;
    item.volume = volume;
    data.push(item);
END %]$data.json\n

Result example:

[{"keyword":"perfume","volume":"50000"},{"keyword":"eyfel perfume","volume":"5000"},{"keyword":"memo marfa","volume":"5000"},{"keyword":"duxi","volume":"5000"},{"keyword":"kenzo intense","volume":"5000"},{"keyword":"climat lancome","volume":"5000"},{"keyword":"v canto","volume":"5000"},{"keyword":"majda bekkali","volume":"5000"},{"keyword":"v canto ricina","volume":"500"},{"keyword":"v canto stramonio","volume":"5000"},{"keyword":"terenzi kirke","volume":"500"},{"keyword":"duhi","volume":"500"},{"keyword":"max mara le parfum","volume":"500"},{"keyword":"stramonio v canto","volume":"500"},{"keyword":"sheikh parfum","volume":"500"},{"keyword":"jacques zolty","volume":"500"},{"keyword":"aj arabia","volume":"500"},{"keyword":"christian lacroix bazar","volume":"500"},{"keyword":"juliette has a gun romantina","volume":"500"},{"keyword":"vilhelm parfumerie mango skin","volume":"500"},{"keyword":"v canto mirabile","volume":"500"},{"keyword":"donna karan dkny be delicious","volume":"500"},{"keyword":"arteolfatto","volume":"500"},{"keyword":"aquawoman rochas","volume":"500"},{"keyword":"angel and demon givenchy","volume":"500"},{"keyword":"venenum kiss","volume":"500"},{"keyword":"v canto mandragola","volume":"500"},{"keyword":"angel demon givenchy","volume":"500"},{"keyword":"hugo boss boss ma vie pour femme","volume":"500"},{"keyword":"nina ricci mademoiselle ricci","volume":"500"},{"keyword":"mmmm juliette has a gun","volume":"500"},{"keyword":"v canto lucrethia","volume":"500"},{"keyword":"mango skin vilhelm parfumerie","volume":"500"},{"keyword":"dalissime salvador dali","volume":"500"},{"keyword":"molecula 02","volume":"50000"},{"keyword":"lucia parfum","volume":"500"},{"keyword":"boadicea pure narcotic","volume":"500"},{"keyword":"terenzi andromeda","volume":"500"}]

tip

More details on outputting results to JSON are described in this article.

Available Settings

note

General settings for all scrapers

Parameter	Default Value	Description
All cookies		Specify all cookies
Cookie "__Secure-3PSID"		Cookie "__Secure-3PSID"
Cookie "__Secure-3PSIDTS"		Cookie "__Secure-3PSIDTS"
Header "x-framework-xsrf-token"		Header "x-framework-xsrf-token"
Url parameter "ocid"("uscid")		Parameter "ocid"("uscid")
Url parameter "authuser"	0	Parameter "authuser"
E-mail		E-mail for Keyword Planner authorization
Password		Password for Keyword Planner authorization
Recovery e-mail		Recovery e-mail
Browser headless (debug auth)	☑	Headless mode for the browser used for login/password authorization
Log Login Screenshot (debug auth)	☐	Create a screenshot of the authorization page and output it to the job log
Date from	Last 12 months	Date from
Date to	Last 12 months	Date to
Language	English	Language
Search networks	Google	Search Network
Currency	USD	Currency
Location code		Location (here you need to specify the location ID, which can be taken from the first column of this table (copy))
Query type	Keyword	Query type
Exclude brand names in results	☐	Brand filter
Exclude adult ideas	☑	Adult content filter
Bulk (packet) mode	☐	Enable batch mode