SE::Google::Trends - Scraper of trending keywords according to Google
Overview of the scraper
Scraper for search queries from Google Trends. The Google Trends scraper can automatically retrieve a list of all categories to which a query belongs, as well as a list of interests by region and time.
You can use automatic query multiplication, substitution of sub-queries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results. Using result filtering you can immediately clean the result, removing all unnecessary rubbish (using negative keywords).
A-Parser functionality allows you to save the parsing settings for the SE::Google::Trends scraper for future use (presets), set a parsing schedule, and much more.
Results can be saved in the form and structure you need, thanks to the built-in powerful templater Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.
Use cases for the scraper
🔗 Date formatting
A way to transform it into a more readable format
Collected data
- List of all categories a query belongs to, in the format (query, category, popularity percentage)
- List of interests over time in the format: time when interested (in Unix format), popularity percentage
- List of interests by region in the format: region, popularity percentage
- List of related rising queries
- List of trending related queries
Capabilities
- Possibility to compare data for several keywords, for this, they must be separated by commas
- Ability to select a country for keyword evaluation
- Selection of the period for outputting results
- Selection of the property for scraping (Web Search, Image Search, News Search, etc.)
- Selection of the Google Trends category (you can find the list of categories using the JS::GoogleTrendsCat scraper)
- Ability to specify an arbitrary date range
- Search by category
Use cases
- Assessing traffic volume by keyword
- Finding new keywords of similar topics
- To set arbitrary dates, you need to select Custom time in Search time and specify the date range in Custom time range in YYYY-MM-DD YYYY-MM-DD format
- Search by category ignores the query provided to the scraper (option Use empty queries(ignore query value))
Spoiler: Example
Queries
Keywords should be specified as queries, just as if they were entered directly into the Google search form, for example:
write essay
Football
Waterfall
Speak in english
Cats and dogs
forex
cheap essay
Query substitutions
You can use built-in macros for automatic substitution of sub-queries from files, for example, we want to add a list of other words to each query, we specify several main queries:
essay
article
thesis
In the query format, we specify the macro for substituting additional words from the file Keywords.txt, this method allows increasing the query variability manifold:
{subs:Keywords} $query
This macro will create as many additional queries as there are in the file for each initial search query, which in total will give [number of original queries (domains)] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro operation.
For example, if the file Keywords.txt will contain:
buy
cheap
As a result, the substitution macro will turn 3 main queries into 6:
buy essay
cheap essay
buy article
cheap article
buy thesis
cheap thesis
Output results examples
A-Parser supports flexible result formatting thanks to the built-in templater Template Toolkit, which allows it to output results in any form, as well as in a structured format, such as CSV or JSON
Exporting the list of related trending queries
Exporting a list of related trending queries in the format: original query, related keyword, popularity.
Result format:
$query:\n$related_queries_rising.format('$key,$rquery,$interest\n')
Example result:
write essay:
write essay,write my essay 4 me,6550
write essay,write essay on globalization and 21st century,1450
write essay,grammarly,1100
write essay,how to write a hook for an argumentative essay,1000
write essay,write my essay for me cheap,1000
write essay,pay someone to write my essay,550
write essay,write that essay login,550
write essay,write my essay cheap,500
write essay,google docs,350
write essay,docs,300
write essay,how to write an argumentative essay step by step,250
write essay,someone write my essay,200
write essay,how long does it take to write a 1000 word essay,200
write essay,how long does it take to write a 2000 word essay,200
write essay,how to write a synthesis essay ap lang,200
write essay,pay to write essay,190
write essay,write essay on my school,180
write essay,how to write essay in hindi,170
write essay,write an essay on my school,170
write essay,help me write an essay,150
write essay,write my essay for me,130
write essay,help me write my essay,120
write essay,write my paper for me,120
write essay,how to write a hook for a persuasive essay,110
write essay,write essay for me,100
Outputting the list of related trending queries to a CSV table
The built-in utility $tools.CSVLine allows creating correct tabular documents ready for import into Excel or Google Sheets.
General result format:
[% FOREACH i IN p1.related_queries_rising;
tools.CSVline(i.key, i.rquery, i.interest);
END %]
File name:
$datefile.format().csv
Start text:
Original query,Related keyword,Popularity
In the General result format, the Template Toolkit templater is used to output the $related_queries_rising array in a FOREACH loop.
In the results file name, you just need to change the file extension to csv.
For the "Start text" option to be available in the Task Editor, you need to activate "More options". In "Start text", we write the column names separated by commas and leave the second line empty.
Saving in SQL format
Result format:
[% FOREACH related_queries_rising; "INSERT INTO serp VALUES('" _ key _ "', '"; rquery _ "', '"; interest _ "')\n"; END %]
Example result:
INSERT INTO serp VALUES('write essay', 'write my essay 4 me', '6550')
INSERT INTO serp VALUES('write essay', 'write essay on globalization and 21st century', '1450')
INSERT INTO serp VALUES('write essay', 'grammarly', '1100')
INSERT INTO serp VALUES('write essay', 'how to write a hook for an argumentative essay', '1000')
INSERT INTO serp VALUES('write essay', 'write my essay for me cheap', '1000')
INSERT INTO serp VALUES('write essay', 'pay someone to write my essay', '550')
INSERT INTO serp VALUES('write essay', 'write that essay login', '550')
INSERT INTO serp VALUES('write essay', 'write my essay cheap', '500')
INSERT INTO serp VALUES('write essay', 'google docs', '350')
INSERT INTO serp VALUES('write essay', 'docs', '300')
INSERT INTO serp VALUES('write essay', 'how to write an argumentative essay step by step', '250')
INSERT INTO serp VALUES('write essay', 'someone write my essay', '200')
INSERT INTO serp VALUES('write essay', 'how long does it take to write a 1000 word essay', '200')
INSERT INTO serp VALUES('write essay', 'how long does it take to write a 2000 word essay', '200')
INSERT INTO serp VALUES('write essay', 'how to write a synthesis essay ap lang', '200')
INSERT INTO serp VALUES('write essay', 'pay to write essay', '190')
INSERT INTO serp VALUES('write essay', 'write essay on my school', '180')
INSERT INTO serp VALUES('write essay', 'how to write essay in hindi', '170')
INSERT INTO serp VALUES('write essay', 'write an essay on my school', '170')
INSERT INTO serp VALUES('write essay', 'help me write an essay', '150')
INSERT INTO serp VALUES('write essay', 'write my essay for me', '130')
INSERT INTO serp VALUES('write essay', 'help me write my essay', '120')
INSERT INTO serp VALUES('write essay', 'write my paper for me', '120')
INSERT INTO serp VALUES('write essay', 'how to write a hook for a persuasive essay', '110')
INSERT INTO serp VALUES('write essay', 'write essay for me', '100')
...
Dump results to JSON
Result format:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.related_queries_rising.key = [];
obj.related_queries_rising.rquery = [];
obj.related_queries_rising.interest = [];
FOREACH item IN p1.related_queries_rising;
obj.related_queries_rising.key.push(item.key);
obj.related_queries_rising.rquery.push(item.key);
obj.related_queries_rising.interest.push(item.key);
END;
obj.json %]
Start text:
[
End text:
]
Example result:
[{"related_queries_rising":{"rquery":["write my essay 4 me","write essay on globalization and 21st century","grammarly","how to write a hook for an argumentative essay","write my essay for me cheap","pay someone to write my essay","write that essay login","write my essay cheap","google docs","docs","how to write an argumentative essay step by step","someone write my essay","how long does it take to write a 1000 word essay","how long does it take to write a 2000 word essay","how to write a synthesis essay ap lang","pay to write essay","write essay on my school","how to write essay in hindi","write an essay on my school","help me write an essay","write my essay for me","help me write my essay","write my paper for me","how to write a hook for a persuasive essay","write essay for me"],"interest":[6550,1450,1100,1000,1000,550,550,500,350,300,250,200,200,200,200,190,180,170,170,150,130,120,120,110,100],"key":["write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay","write essay"]}}]
For the "Start text" and "End text" options to be available in the Task Editor, you need to activate "More options".
Results processing
A-Parser allows processing results directly during scraping, in this section we have provided the most popular use cases for the SE::Google::Trends scraper
Using an arbitrary date
To set arbitrary dates, you need to select Search time in Custom time and specify the date range in Custom time range format in YYYY-MM-DD YYYY-MM-DD.
Download example
How to import the example into A-Parser
eJx1U1GP2jAM/iuVxcNN4hDctJe+cWhMm9j1dsATQqeIuCy7NMmSlA1V/e9z0tIC
46RKjZ3vsz/HdgWeuTf3bNGhd5BuKjDxDCksP6fpF633EtN0ZVFxl9wna4fJrnRe
FwlnHmEIhlmHNnA3NygE4JizUnoYVuCPBimyPqC1gge24GQ7ZHb389WLIrgOTJYB
1aSB+n1eg7jmPYwfxvfjCX0JHen/kY5Qb7dDoNJIiZtrW7BQ4sBMRm293eWSHXCl
6TIXEnv3nKwnVoQEg1B5uB3lMdDdh5H/GyIwzoUXWjHZZAiP0mddK/E7ClSasHS0
At3cUokpeIwBgvN4UreBQbSBQpSR+6PhQJoz6XAIjqTOGQnh1zfCo2Ve28wEPeSv
QKuplAs8oOxhMf5jKSSnDk5zIn1tibch2X8x6q6881TUpz+WNHRRovWYfe9ZXC/0
/vQYUhTCk+1mulShMWNyviGa7s2eAqzQFrs0beQ2O02woXkjZN+yqeldF2VctOXS
udMqF/usHbQTslQrWpNMzXRhJIa6VCkltcXhSz8eU9e2IRi9wGvyLKYgWd1igNda
um/LRqqxgsbvUxBY0EueZ21D7piU65fF+Q30I0VGfO4EnWNHCHCPe02DRMXU225j
u6Wvbu1tWtXUpV/uuQGHkqKT9oxMagGkk/ofqyh6IQ==
Filtering results
Example of exporting a list of related trending queries whose popularity is below a specified value.
Download example
How to import the example into A-Parser
eJxtVMFu2zAM/ZVACNAVcIP20IsPA9JgKTpkddekpzQohIoOtMiSK8pZA8P/PkqW
7aTLzaRIvsdH0jVzHHf4ZAHBIUvXNSvDN0vZ8kea3huzVZCmKwta4OhqNJfKgR3B
Jy9KBSxhJbcI1qeuz2RQgICcV8qxpGbuUAIVNnuwVgqfLQXZubEFdwQbwtieq8qH
jT8qsIf09VWPLSjuQLx5jwR8sxKl3k7axG8X4x0ckrEN8clYamII6Cjx4pI1A24e
uBNChErX7HxhTywWYZsk5q3aIt+PGN5eX5NlSieNJhNBI2s2mw4A54Gfb6W8mURd
+8cl38PKtLRgcJPA8MiLIIAgbv61a/Ry4j59BS6E9JhctQhe/QH1RcuPQE8bio2d
za0pyOUgFAhKdezWUWnfahVyf7c5LM25QkgYEtU5JyLi64skYbgzNgsakL9mRk+V
WsAe1BAW6t9VUglalWlOSQ8x8XxI9l+Npm/vGIoW6a8lDn2VYN1lv4YsYRZm24mh
ZCEd2TgzlfaD8fPbAZS9Zo8+rDAWephYOaLTpZS02BQ5jGxaDq6TNk7Gcup8NzqX
2yxeQhdZ6RWdY6Znxl+X70tXStFYEJ6H9ZhiHIM3BoJfk2cBgmj1F8icMQp/Lluq
pZW0freeYEFKHqPGku9cqZfnxfELG1aKjCD3CBD5gflwB1tDi0TNNJv+19D/XOpz
P4i0bmhKf/CpDfYtBSddGZnhsG6af/TaovA=
See also: Result filters
Possible settings
| Parameter name | Default value | Description |
|---|---|---|
| Search region | Worldwide | Select region |
| Language | English | Select language |
| Search time | Default(Past 5 years) | Select the period for outputting the result |
| Custom time range | Empty string | Enter an arbitrary date range when Seacrh time = Custom time (format YYYY-MM-DD YYYY-MM-DD) |
| Search category id | 0 | Select category |
| Search property | Default(Web Search) | Select property for scraping (Web Search / Image Search / News Search / Google Shopping / YouTube Search) |
| Use empty queries(ignore query value) | ☐ | Ignores the query value supplied to the scraper and searches by category |