Rank::Ahrefs::KeywordGenerator - Ahrefs Keyword Generator Scraper
Overview of the scraper
Parsing thousands of keyword variations in seconds. Finding relevant keywords from Ahrefs' database containing over 8 billion queries.
The scraper collects keyword ideas and keyword questions. The data is collected from the page https://ahrefs.com/keyword-generator. The scraper allows selecting a country and selecting search engines (Google, Bing, Youtube, Amazon).
To run the scraper, you need to select a preset for captcha solving 
Util::Turnstile.
Collected data
- Keyword ideas
- Keyword questions
- Keyword Difficulty (KD) - this value assesses how difficult it is to rank in the top 10 organic search results for a keyword, on a 100-point scale.
- Frequency - this metric shows the average number of times people in that country search for your target keyword per month.
- Date of the last check of search engine results for the keyword.
Capabilities
- Automated work with proxies
- Select the country for which keyword difficulty will be determined
- Select the search engine for which keyword difficulty will be calculated
Use cases
- Retrieving summary information about a keyword query
- Assessing the frequency of a keyword query
- Assessing keyword difficulty
Queries
Keywords must be specified as queries, for example:
essay writing
buy notebook
buy ticket
If the queries are in Russian or any other language, choose the corresponding country in the scraper settings. This is because there might be no results for the country United States, which is set by default.
Output results examples
A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured format, such as CSV or JSON
Outputting keyword ideas and questions to results
Result format:
Ideas:\n$ideas.format('$key\n')\nQuestions:\n$questions.format('$key\n')
Example result:
Ideas:
best buy notebook
best buy notebook computers
buy notebook
buy notebook computer
essay writing service
essay writing
best essay writing service
essay writing services
essay writing help
buy airplane ticket
buy train ticket
rzhd buy ticket
buy rzhd ticket
aeroflot buy ticket
...
Questions:
where to buy midori traveler's notebook
where to buy moleskin notebook
where to buy moleskine notebook
where to buy dot grid notebook
where to buy traveler's notebook
a common mistake when writing a reflective essay is to
how to start writing an essay
how to improve essay writing
what is john steinbeck’s purpose in writing the essay "symptoms"?
what is the second step in writing an argumentative essay
how many days in advance can I buy a train ticket
how to buy an airplane ticket
how to buy an electric train ticket
where to buy a lottery ticket
how to buy a train ticket
...
Outputting 'keyword ideas' with parameters (difficulty, frequency, and check date) to a CSV table
The built-in utility $tools.CSVLine allows creating correct tabular documents ready for import into Excel or Google Sheets.
Result format:
[% FOREACH item IN ideas;
tools.CSVline(item.key, item.difficulty, item.volume, item.updated);
END %]
File name:
$datefile.format().csv
Initial text:
Key,Difficulty,Frequency,Date
Example result:
Key,Difficulty,Frequency,Date
"essay writing service",98,7500,2021-03-04T12:43:40Z
"essay writing",90,6600,2021-03-04T11:05:22Z
"best essay writing service",96,3000,2021-03-03T19:37:02Z
"essay writing services",98,900,2021-03-03T02:39:34Z
"essay writing help",91,700,2021-03-05T07:13:18Z
"best essay writing service reddit",49,600,2021-03-03T22:07:33Z
"cheap essay writing service",90,600,2021-02-26T01:25:25Z
"essay writing tips",64,600,2021-02-27T22:15:00Z
"custom essay writing",87,450,2021-02-20T03:51:14Z
...
Download example
How to import the example into A-Parser
eJyFVF1v2jAU/SuRRaVWYtE6aS/ZU5rB1o2RNpS9MIS85IZ6OLZnO7QI8d937YQE
ui9eyP0691zfY++JpWZj7jQYsIZEiz1R/ptEZPRMK8VhFT9qKE0UfYbdk9TFBxCg
qZWaDImi2oB2dQuSUbGJon8kF1DSmlsy3BO7U4Ad5Ba0ZgVgkBVozy3jqwwSqmz+
SN+sWipDsqW8dgUZ5FRh6JWm3yX+7yg5/B2ulLqiFjF81w5kcRGM02wUJx8DZqEK
bqcBFlHz7psI/M9KyU2YzL5yJuDS5YQb2A19dliwsmQ5Ih4dW8nrClqjVgW1UFw1
WKPp++BiSQ7L5ZA0NMzYc0IWA3UddvO1wRndwoN0zBmH3j1Ga0orx33g4F00bIa7
vArts0OgRcEsk4LypoPbSd91LthPP7uQmIufmoEZa1mhy4IHcM7dkd2CDLxNEKL2
tfdNDYlKyg0MiUGqY4pEipcRPAe/8VQ5PujfEyliziewBd6nefybmvECBRSXWHTb
Fv45Jf0N49CNd9oKNfCkkUOH4q2b9EtfVciJXB8Pg7OKWbRNImvhFvManRsA1Z3Z
1KVVUkPXpkVuu+PlUSCc4PqVxap3nY1xtpZzZy5FydZpK+JjZi0e8IamIpHuMrq5
RM05rsVA1ssjNu0anNETfFmc+BZIq7uLxIv906yhqjRD+b11BCs8ydOuLWROOZ9n
k9MI6SXl5WQcbI46XUtUEE5xWHYvRffQ7P/3XkT7A67qh7lrCt1crgx9eEAG90Ci
68MvqRKrIg==
In the Result Format, the Template Toolkit templating engine is used to output the elements of the $ideas array in the FOREACH loop.
In the result file name, you simply need to change the file extension to csv.
For the 'Initial text' option to be available in the Job Editor, you need to activate 'More options'. In 'Initial text', write the column headers separated by commas and make the second line empty.
Outputting 'keyword questions' with parameters (difficulty, frequency, and check date) to a CSV table
The built-in utility $tools.CSVLine allows creating correct tabular documents ready for import into Excel or Google Sheets.
Result format:
[% FOREACH item IN questions;
tools.CSVline(item.key, item.difficulty, item.volume, item.updated);
END %]
File name:
$datefile.format().csv
Initial text:
Question,Difficulty,Frequency,Date
Example result:
Question,Difficulty,Frequency,Date
"a common mistake when writing a reflective essay is to",0,150,2021-03-04T12:17:36Z
"how to start writing an essay",58,150,2021-02-24T09:48:50Z
"how to improve essay writing",24,150,2021-02-28T01:16:10Z
"what is john steinbeck’s purpose in writing the essay ""symptoms""?",0,150,2021-03-01T20:33:45Z
"what is the second step in writing an argumentative essay",1,150,2021-02-21T21:36:03Z
"when assembling a writing plan for an essay, which of the following should come first?",0,150,2021-02-16T15:20:21Z
"which step must you complete before writing a media assessment essay?",3,100,2021-03-04T02:48:14Z
"how is theme treated when writing a response to literature essay?",1,100,2021-03-01T13:54:31Z
"which phase of a writing strategy involves deciding on the topic and the purpose of the essay?",0,100,2021-02-19T23:58:36Z
...
Download example
How to import the example into A-Parser
eJyFVN9v2jAQ/lcii0qtxKJ10l6yJ5rB1o1BB2UvFCFDLtTDsV3boY2i/O87OyGB
btV4IXe+77vvftglsdTszZ0GA9aQaFkS5b9JRIYvNFMc1oNHDamJou9QPEudfAEB
mlqp1085GMukMKRPFNUGtGNYkhkV+yh6C4bBCaQ055b0S2ILBZhLHkBrlgAesgTt
hWV8PYOYKrt9pB/Wjag+OVCeO8AMtlTh0TtNNxL/C0qqt+lSqTNqkcNnbUmWF0Ew
ms6Gg/hrwCxkwe0kUNdhW9enBxHUPyslN2E8/8WZgEsXG+6h6HtUmLA0ZVukPjoO
kucZNEauEmohuWrIhpPPwcWKVKtVn9SCzMirQz09TN5W2hzO6QHupauBcejcI7Qm
NHNV9By/Ow3rMi+vQvviGGiSMFcH5XUGN50u60KwJ98FITEWPzUDM9IyQ5cFT+Cc
xVHdkvS8TZAi99ifNYZEKeUG+sSg1BFFIcnrE2yEn/1U1fsSlUSKAedjOADvwjz/
Tc54gqs0SBF02wD/HTL9i6NqyztNhdvwrFFDy+Ktm+mPDpXIsdwdm8FZxizaJpa5
cIN5j849gGp7NnFhmdTQpmmYm+x4oRQIt3rdyAaqc52VcTaWc+dWipTtps06HyNz
cY+3dipi6S6oq0vknONYDMy69RiYZgzO6AS+Bsc+BcpqbyXx2/5tXktVmuH6fXQC
M+zkadaGcks5X8zGpyekWyk0wBhaBK7pTOwexCYvAiEtbKTcEwe3sJO4WFhctWqf
kvZNKv/3oERlhRP8be5qoCvXwdCHfTM4HhJdV38AHCm7pA==
In the Result Format, the Template Toolkit templating engine is used to output the $questions array in the FOREACH loop.
In the result file name, you simply need to change the file extension to csv.
For the 'Initial text' option to be available in the Job Editor, you need to activate 'More options'. In 'Initial text', write the column headers separated by commas and make the second line empty.
Dump results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.query = query;
obj.questions = [];
FOREACH item IN p1.questions;
obj.questions.push(item.key);
END;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example result:
{"query":"casino online","questions":["online casino no deposit bonus keep what you win","casino ohne anmeldung gratis online spielen where& 39","why online gambling is more dangerous than casino gambling","how to start an online casino","online casino no deposit bonus keep what you win usa","online casino games where you can win real money","how to open online casino","how to start an online casino business","when will the casino open in gta 5 online","how to play online casino games","what is the best online casino","what online casino has gammomat provider for canadians","online casino no deposit bonus keep what you win usa 2017","what online casino has the best payouts","how to start an online casino free","what online casino pays out the most","which online casino pays out the most","how to cheat online casino slot machines","how do online casino slots work","what is the best online casino canada","how to create online casino business","where can i play casino games online for free","what is the best online casino to play at","which online casino is legit","how to win money online casino","how to play online casino","how to win online casino roulette","what is the best online casino game","what is the best online casino for slots","what is the best online casino yahoo answers","how to start a online casino","how to play casino games online","which online casino game has the best odds","how to win online casino slots","how to open online casino business","what online casino is legit","usa online casino no deposit bonus keep what you win","how to buy an online casino","where can i play free casino games online","what is the best online casino for roulette","how to withdraw money from online casino","how to make money from online casino bonuses","what is the best online casino for us players","what is the best bank or credit union to have for online casino deposit","how to start casino mission gta 5 online","how to pay taxes from online casino reddit","how does online casino work","what other online casino is affilated with royal planet casino","where do you enter little casino promo codes online","what is harder play casino or online poker"]}
For the 'Initial text' and 'Final text' options to be available in the Job Editor, you need to activate 'More options'.
Results processing
A-Parser allows processing results directly during scraping. In this section, we provide the most popular use cases for the Rank::Ahrefs::KeywordGenerator scraper
Keyword ideas deduplication and keyword questions
You need to add deduplication twice and select from the dropdown lists:
$questions.$i.key - Keyword and $ideas.$i.key - Keyword.
Download example
How to import the example into A-Parser
eJydVN9vGjEM/leQhdRVYmidtJe8UTSqbqy0FJ4AocD5aEYuCUmO9oT43+vcT2jX
TdoLnO3P3+fYTg7gudu6e4sOvQM2O4DJv4HB9xeeGInL3pPF2DH2E7NnbaMbVGi5
13a5W4plqsQOOmC4dWgDwQzGXG0Z+yiLwBHGPJUeOgfwmUGS0nu0VkRIQRGRPfVC
LsfY58avn/jXZVlTB/ZcpiFhjGtuKPTZ8pWm/4zD8WO6WNuEe+LIVWuS2wi5Y/O5
aovw1S1gny7aW8zIe3FJPw8pOi+0KnC7ynqPPdUPTUmDeinJZpArkIfgsOiUiEkB
d94KtYHKeyP1iktg3qb4d9K6nP8gXiwqJjfIz0L4trnq1q0ug498jxMdmihko+4G
ZN3xJKi0I+4xRKumXHb9S2DgUSRCeVwWCmE9GtVpcR4GShOWPq1AN7A6IZfHnCA4
s6q6GbRzuznlQ5EDLObSYQcclTrgVEj0NiJ8sXwjU7SLHUCrnpRD3KNsYDn/dSpk
RLvciynptkz8M2T0juNYH+9Uitbx2VINNUtuXY9+NVmRHupN1QwpEuHJdn2dqjCY
L/mA0dQ9uwuwRFusZUrmUp0utEEVdr8ZWc80rrNjnI3l3LnWKhabUXmfKmSqJvRq
jFRfhwcinEulUtJYHI6b9ei5cgzBaAp8m9zPJais+lkAr7V0Px6LUo0VtH7fQoEJ
dfJUtaRccymn4+FpBJqVIgOd41krNJ2uw1yt0qyltMeV1tu5qr5aZk0hulB1rOV1
i7AQFDxuNO0enf+4qJ+7+tk8/OvRY4cjDfm3uy8SQ0dCGvmotY4mCOzq+ApW0Pkz
See also: Results deduplication
Filtering questions with keywords
Add a filter and select from the dropdown list: $questions.$i.key - Keyword. Select type: Contains string. Next, in String specify the filtering criterion. For example, filter questions that contain the word what.
Download example
How to import the example into A-Parser
eJyFVFtv2jAU/ivIQuoqMbRO2kveKBrTNlZaCk+AkCEn4OHYxnZoI5T/vnOcG3Sr
9pScy/edu8/Mc3dwjxYceMeixZmZ8M8i9vWVp0bCerC3kLgo+gn5i7bxN1Bgudd2
fVwnQnrLesxw68ASfsGmXB2i6D0QOseQ8Ex61jsznxvASPoE1ooY0ChilOdeyPUU
htz47Z5/Xlcp9diJy4wAU9hyg6aPlm80fnPOivfpEm1T7pEjRG1InjJwXmjlouVS
dY+11C/dP9x0D5Cj5eb2kpsqBqqioosWrEGiFiFs1au8ZiVkq5XnQl1EftlzSkQb
gqEslAOEF6tVzetGIQm0dc1dv6m/Mj7zE8x0mQ206hFKDzylCN2YeyBrXc1t378S
A49jQVG5LCPQzNqocyWOIUOl0Rd/rQA3sjpFlYdAQMq8zm7BukGmmrOAfSoxLEq4
dNBjDlMdcUwkfmsRvtyIiSmbF52ZVgMpx3AC2boF/vtMyBgXbJAg6HsF/LfL5C+O
oinvMhTuyIvFHBqWIN1PfrWoWI/1rm6GFKnwKLuhzhQN5lMYN5imZw/klmoLTZiK
uYqOR2ZA0UK2IxuYVnVVxtVYrpW4T4nYTaolrz0zNcNLnqihpqOlulQmJY7FwbRd
j4GrxkBCm+Bb8DCEwLSaW2Vea+l+PJepGitw/b5Qgil28jJqRbnlUs6n40sLa1cK
BXCO5x1qulC7pdpkeUdpDxutD0tV/3XMFk14Xo2t43UHfRlF8LDTuHtYf7Fq3qDm
KTv/7yWKzgUO+bd7LIHUEYKhDlvrwmXeFX8Al3PStA==
See also: Results filters
Possible settings
| Parameter | Default value | Description |
|---|---|---|
| Util::Turnstile preset | default | Preset selection Util::Turnstile for bypassing captchas. You must first configure the Util::Turnstile scraper, specify your access key and other parameters, and then select the created preset here. |
| Turnstile pass proxy | ☐ | Pass proxy to the solve service. Details. |
| Search engine | Google | Search engine selection |
| Country | United states | Search country selection |
| Do not search for sitekey | ☑ | Experimental option, disables searching for sitekey captcha, which in turn speeds up job startup. If there are problems such as an invalid sitekey, this option should be disabled. |
| Additional headers | Option to specify arbitrary request headers |