SE::Google::Suggest - Google search suggestions scraper
Overview of the scraper
Scraper for search suggestions by keywords in Google. Thanks to the SE::Google::Suggest scraper, you can automatically collect keyword bases from Google search engine suggestions by query. Using the SE::Google::Suggest scraper, you can easily and quickly scrape Google suggestions by query based on the selected country, language, or domain.
The Google suggestions scraper solves one of the main SEO tasks, namely the rapid automated obtaining of an extended semantic core. Google search suggestions allow you to cover the maximum number of phrases, and in combination with the Google Keyword Planner scraper - 
SE::Google::KeywordPlanner you will get the most complete semantics, which will help attract more organic traffic.
Thanks to the multi-threaded work of A-Parser, the processing speed can reach 6000 requests per minute, which on average allows obtaining up to 45000-46000 results per minute.
You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to obtain the maximum possible number of results. Using result filtering, you can immediately clean the result by removing all unnecessary junk (using negative keywords).
A-Parser functionality allows you to save the scraper settings of SE::Google::Suggest for further use (presets), set a scraping schedule, and much more.
Saving results is possible in the form and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
Use cases for the scraper
🔗 JS::Google::FromSnippets
Example of work in JS-scrapers with built-in scrapers
Collected data
- Number of results by query
- Suggestions for the query
- Type of suggestion (0 - human, 1 - artificial)
Capabilities
- Supports choosing the country of search, domain, language of results
- Ability to choose the source that needs to be used to collect suggestions (option Client)
- Ability to remove HTML tags from parsing results (option Remove HTML tags)
Use cases
- Collecting keyword bases
Queries
As queries, you need to specify search phrases, for example:
write essay
Football
Waterfall
Speak in english
Cats and dogs
forex
cheap essay
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add some list of other words to each query, let's specify several main queries:
essay
article
thesis
In the request format, we will specify a substitution macro for additional words from the Keywords.txt file, this method allows to increase the variability of requests many times over:
{subs:Keywords} $query
This macro will create as many additional requests as there are in the file for each original search query, which in total will give [number of original requests (domains)] x [number of requests in Keywords file] = [total number of requests] as a result of the macro's work.
For example, if the file Keywords.txt contains:
buy
cheap
As a result, the substitution macro will turn 3 main queries into 6:
buy essay
cheap essay
buy article
cheap article
buy thesis
cheap thesis
Output results variants
A-Parser supports flexible formatting of results thanks to the built-in Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Export of suggestions list
Result format:
$results.format('$suggest\n')
Example of result:
buy essays online
buy essay cheap
buy essay uk
buy essays online no plagiarism
buy essay papers
buy essay online reviews
buy essays reddit
buy essay friend
Output of query, query suggestions + type of suggestion
Result format:
$query:\n$results.format('$suggest - $type\n')
Example of result:
cheap essay:
cheap essay writing service - 1
cheap essay writing service uk - 1
cheap essay writing service canada - 1
cheap essay writing 24 - 1
cheap essays online - 1
cheap essay writing service reddit - 1
cheap essay writing service australia - 1
cheap essay writing service review - 1
buy essay:
buy essay online - 1
buy essay cheap - 1
buy essay uk - 1
buy essay papers - 1
buy essay online reviews - 1
buy essays reddit - 1
buy essay friend - 1
buy essay online uk - 1
Output in CSV table
The built-in utility $tools.CSVLine allows creating correct table documents, ready for import into Excel or Google Sheets.
General result format:
[% FOREACH i IN p1.results;
tools.CSVline(i.suggest);
END %]
File name:
$datefile.format().csv
Initial text:
Подсказки
In the General results format, the Template Toolkit is used to output the elements of the $results array in a FOREACH loop.
In the file name of the results, you just need to change the file extension to csv.
To make the "Initial text" option available in the Task Editor, you need to activate "More options". In the "Initial text", we write the names of the columns separated by commas and make the second line empty.
Keyword competition
Saving in SQL format
Result format:
[% FOREACH results; "INSERT INTO serp VALUES('" _ query _ "', '"; suggest _ "', '"; type _ "')\n"; END %]
Example of result:
INSERT INTO serp VALUES('cheap essay', 'cheap essay writing service', '1')
INSERT INTO serp VALUES('cheap essay', 'cheap essay writing service uk', '1')
INSERT INTO serp VALUES('cheap essay', 'cheap essay writing service canada', '1')
INSERT INTO serp VALUES('cheap essay', 'cheap essay writing 24', '1')
INSERT INTO serp VALUES('buy essay', 'buy essay online', '1')
INSERT INTO serp VALUES('buy essay', 'buy essay cheap', '1')
INSERT INTO serp VALUES('buy essay', 'buy essay uk', '1')
INSERT INTO serp VALUES('buy essay', 'buy essay papers', '1')
...
Dumping results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.totalcount = p1.totalcount;
obj.suggests = [];
FOREACH item IN p1.results;
obj.suggests.push(item.suggest);
END;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example of result:
[{"suggests":["buy essay online","buy essay cheap","buy essay uk","buy essays online no plagiarism","buy essay papers","buy essay online reviews","buy essays reddit","buy essay friend"],"totalcount":8}]
To make the "Initial text" and "Final text" options available in the Task Editor, you need to activate "More options".
Results processing
A-Parser allows processing results directly during scraping, in this section we have provided the most popular cases for the scraper SE::Google::Suggest
Parse to level option
The option tells the scraper to substitute the obtained results into the query queue to the specified depth level, for example:
- If the 1st level is specified, the scraper will add to the queries all the results obtained from the original query
- If the 2nd level is specified, the scraper will add to the queries all the results obtained from the original query + all the results obtained from the 1st level queries
- and so on.
In simple terms - this is substituting the obtained results into the query queue, which in turn allows scraping more results. Since there will likely be duplicates in the results, to prevent the scraper from doing extra work (not scraping the same thing), it is recommended to enable unique queries (Unique queries)
Download example
How to import an example into A-Parser
eJx9VE1v2zAM/SsGkcMGZEF72MW3NFiGDlmd1c0pyEGIaVerLHmSnDQw/N9HKo7t
bEFvpvj4+PFIN+CFe3Nriw69g3jbQBW+IYb0Wxx/N6ZQGMdpXRTofPQlWgvrMPIm
UnhAFeG7KCuFMIWKHZY5trdCCZFhLmpFXw34U4WUwlReGu3IJzMyA8WKeenlIFRN
mPu7doQ/oLUyw0tAZkoh9QCG4/E4K0Li2d6U8EFoMcoBm/Qj6OsYihra3W4KNCXq
xS2NLQVPa1Ldz7rR9c5UHPDFkDOXYUSXGLKeRMlsk0x4ZO8sD0SfPs/8OzOILJM8
G6HOGXiuQ9aNln9CNc5bqQvCk2kluqWlrmPwGEj48XSpcAuTYAPR1CH+1zkGYm9r
nIKjapeCasl6Ry6UI4/0aIU3Nunkihsweq5UkGqABfqHWqqM9mCeU9BjF3gbkvzH
0fYdjlORHEdLNfQswXpIfg5RmVmZghrXhtpWspSebLcwtWZt7ujxDbHqx/bEsNJY
7NN0zF12uocKNWs/qDavhqerNq6UuX7cG53LIun26YKs9QsdXaIXhk+H+9K1UqSK
w+dhQ+auk4GNocB/gxchBd/C5brAG6Pcj/RcamUlbeBXLrCkSY6zdpR7odTmeTX2
wLBRfFQ87gidEydguMfC0B5RM+2uv/v+F9LcvP64aUmm3259RnNPjKU3Go4jDejS
27/f2JZm
Filtering results (using negative keywords)
By using negative keywords, it is possible to immediately remove results that you do not need. Similarly, by using a filter, you can also keep only those results that contain the necessary words.
Download example
How to import example into A-Parser
eJx9VE1vGjEQ/SvIQkojpSgcetkbQaFqRUMa4EQ4WOywceO1tx4vFBH+e2e83g9a
xG1n5r2ZNx/ro/AS3/HZAYJHkayOogjfIhHzxyT5am2mIUnmZZYB+t7n3hKhlytT
Ym9vXYriThTSITgmry5xCJHCVpaavo7CHwqg3Lbwyhpmq5TMkGIKO9Dk2UldEmZ4
f+rgd+CcSqEmpDaXyrRgsd/vB1koPNjYXFyhZp0aYjm/Bn3rQsF0oVulPTgK07S4
tWQVv7gnjJ2v7yJuUZEcGOtz6TdvnbwO0lT5D6U1ZFJ/YCGNQgZUMyLENYlb6yhh
FNEm7Ucxgyr+6aYfNb2+mptbcVqva+U4CQjmFMNBXH4TnMsdLGzVL7TuCVlPMg+V
UumBo3Wp24H/wxkkt0UNSF1V4ANpqy6N+h2UonfKZIQn0ynAiaP1JcJDSMLOQ61w
JfrB5rmWgf+z4ojEuxJo7qR2IklL2gS2UiNFFC1Beutm8e6So7BmpHW4uRYW0j+U
Sqd00KMtkb5F4mXI7L8cp6bDbila2t6RhlpoMB5mP1pSaqc2o76Npa61ypUnG8e2
NLyae3K+AxTN1J4YllsHTZUoLxanH7oAwwfSLm1UtK6zLs4Wc+7cWLNV2SweXY0s
zYJejZkZ27zQwG2ZUmtaCsJLeyAjjFtgoxX4L3kcSvA/Xb8Swlur8fu8klo4RQf4
hQXmNMhu1ZhyI7Vevky7EdEeFD8OPO4eIMqDYLiHzNIZUTOndfN+NW/g8eIrlhxP
tKZf+FyhuSfGko+Gg+FHHZ7+As4s2Yc=
See also: Results filters
Possible settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Client | Chrome omnibox | Choice of the source to use for collecting suggestions (Search page / Chrome omnibox) |
| Follow suggests | Human | Choice of the type of suggestions to insert into queries when using Parser to level (All / Synthetic / Human) |
| Google domain | www.google.com | Domain selection |
| Search from country | Global | Country selection from where the search is performed |
| Interface language | English | Interface language selection |
| Remove HTML tags | ☑ | Removing HTML tags |