Rank::Ahrefs::KeywordDifficulty - Ahrefs Keyword Difficulty Checker Scraper
Overview of the scraper
Rank::Ahrefs::KeywordDifficulty – This scraper retrieves the "Keyword Difficulty" value. It also retrieves additional data for the search results for the query (only for the first page of TOP - 10).Keyword Difficulty is a metric that helps understand how difficult it will be to rank for a specific search query.
Data is collected from the page https://ahrefs.com/keyword-difficulty.
Requires connection to a recaptcha solving service 
Util::Turnstile.
Collected data
- Keyword difficulty
- Estimated number of websites from which backlinks need to be placed to enter the TOP10 for the target keyword (shortage)
- Overview of search results for the query (only the first page of TOP - 10)
- Links from search results (TOP - 10)
- Anchor (TOP - 10)
- AR (TOP - 3 only)
- DR (TOP - 3 only)
- UR (TOP - 3 only)
- Backlinks - the number of backlinks from other websites leading to this search result (TOP - 3 only)
- Referring Domains - total number of unique domains linking to this search result (TOP - 3 only)
- Traffic - estimated monthly organic traffic received by this search result (TOP - 3 only)
- Keywords - the number of keywords for which this search result ranks in the top 100 positions (TOP - 3 only)
Capabilities
- Automatic proxy handling
- Select the country for which keyword difficulty will be determined
Use Cases
- Obtaining summary information about the search results for a query from the TOP-10
- Assessing the difficulty of a keyword/query
Queries
Keyphrases should be specified as queries, for example:
essay writing
buy notebook
buy ticket
tire shop St. Petersburg
If the queries are in Russian or any other language, select the corresponding country in the scraper settings. Otherwise, there might be no results for the country United States, that is set by default.
Output results examples
A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in structured formats, such as CSV or JSON
Outputting the "query difficulty" in the result
Result format:
$query: $difficulty\n
By default, the query and the keyword difficulty value will be displayed in the result:
essay writing: 84
buy notebook: 80
tire shop St. Petersburg: 55
buy ticket: 90
Outputting additional parameters for the top-10 keyword search results to a CSV table
The values anchor, ar, dr, ur, bl, domains, traffic, keywords are available only for TOP-3.
The built-in utility $tools.CSVLine allows you to create correct tabular documents ready for import into Excel or Google Sheets.
Result format:
[% FOREACH item IN serp;
tools.CSVline(query, difficulty, item.link, item.anchor, item.ar, item.dr, item.ur, item.bl, item.domains, item.traffic, item.keywords);
END %]
File name:
$datefile.format().csv
Initial text:
Query,Difficulty,Link,Anchor,AR,DR,UR,BL,Domains,Traffic,Keywords
Example result:
Query,Difficulty,Link,Anchor,AR,DR,UR,BL,Domains,Traffic,Keywords
"buy notebook",80,https://www.bestbuy.com/site/laptop-computers/all-laptops/pcmcat138500050001.c?id=pcmcat138500050001,"Laptops and Notebooks: PC Laptop, Notebook - Best Buy",2120,88,34,219,80,139112,2248
"buy notebook",80,https://www.bestbuy.com/site/computers-pcs/laptop-computers/abcat0502000.c?id=abcat0502000,"Laptops & Notebook Computers - Best Buy",2120,88,49,2306,474,37222,3148
"buy notebook",80,https://www.bestbuy.com/site/searchpage.jsp?_dyncharset=UTF-8&browsedCategory=pcmcat138500050001&cp=2&id=pcat17071&iht=n&ks=960&list=y&sc=Global&st=categoryid%24pcmcat138500050001&type=page&usc=All%20Categories,"Laptops and Notebooks: PC Laptop, Notebook - Best Buy",2120,88,14,2,1,243,179
"buy notebook",80,https://www.amazon.com/Notebooks-Laptop-Computers/b?ie=UTF8&node=565108,"Laptops | Amazon.com",,,,,,,
"buy notebook",80,https://www.walmart.com/cp/laptops/1089430,"Laptops - Walmart.com",,,,,,,
"essay writing ",84,https://owl.purdue.edu/owl/general_writing/academic_writing/essay_writing/index.html,"Essay Writing // Purdue Writing Lab - Purdue OWL",1256,90,45,731,316,2702,864
"essay writing ",84,https://grademiners.com/,"Essay Writer & Essay Writing Service | GradeMiners.com",7081,82,87,390507,21505,7328,2836
"essay writing ",84,https://www.scribbr.com/category/academic-essay/,"The Beginner's Guide to Writing an Essay | Steps & Examples",144207,66,40,966,249,12199,3281
"essay writing ",84,https://www.internationalstudent.com/essay_writing/essay_tips/,"General Essay Writing Tips - Essay Writing Center",,,,,,,
"essay writing ",84,https://writingcenter.fas.harvard.edu/pages/essay-structure,"Essay Structure | – Harvard Writing Center",,,,,,,
...
"buy ticket",90,https://www.tutu.ru/,"Tutu.ru: Air, Rail, bus tickets and tours online ...",39974,74,80,459414,2349,1052237,7272
"buy ticket",90,https://www.rzd.ru/,"Passengers",11185,80,84,5118775,7165,4163730,32342
"buy ticket",90,https://pass.rzd.ru/,"Passengers",11185,80,75,475270,1426,249503,9533
"buy ticket",90,https://www.aviasales.ru/,"Cheap flights online, ticket search for ...",,,,,,,
"buy ticket",90,https://www.kupibilet.ru/,"Cheap flights online, search for plane tickets ...",,,,,,,
"tire shop St. Petersburg",55,https://www.spbkoleso.ru/,"Buy wheels, discs and tires in the online store ...",4527548,28,42,1489,285,2240,855
"tire shop St. Petersburg",55,https://spb.kolesa-darom.ru/,"«Kolesa Darom» — online store for tires, wheels and ...",370176,55,27,136,44,1740,146
"tire shop St. Petersburg",55,https://www.koleso-russia.ru/,"KOLESO - car tires and wheels | car tires | car rubber ...",992706,44,40,1295,222,14428,1319
"tire shop St. Petersburg",55,https://kolesa812.ru/,"Tires and wheels in Saint Petersburg | Buy in the online ...",,,,,,,
"tire shop St. Petersburg",55,https://ship-ship.ru/,"Online store for tires and wheels - buy cheaply ...",,,,,,,
...
Download example
How to import the preset into A-Parser
eJyFVNtOGzEQ/ZWVBRJI24gAVdH2KYREpaUJDdCXEEVmPQtuvLaxvYEoyr937L0l
0MuLd25n5sx4vGviqF3YawMWnCXJdE10kElCBq801wLmvScDmf0GqwueZTwthFuR
mGhqLBgPmZIJlYskKeOSBCNflGE70QwyiiKJ18StNGB2tQRjOAN0cob6neNiPoE+
1S59osfzikZMllQUHjCBlGp0fTD0QeF3Rcnm7+lSVUhnVlt4U/wrPlMmpw5rBpYN
aLofDceTQa//JeIO8uhyFOluBxvXn+9lFEVOKWE7/Zufgks4eC7ArOKINZ3HAdVB
56ISqUyflKmVWmC1UNTCg6hdKqdc2kpzhvrclbYoJ20PkcxgdBHtz8hmNotJ2YYd
hp6wiz3k3Myzct7QJdwq3zkX0JqHqI1o7nvfY9SB93bK4Rwcdtyrz0AZ444rSUVZ
wS9BW/VO8ucwO6kw1o+Egx0alaPJQUgQ5lSzm5K9oBNMUQTsjxJDkowKCzGxSHVI
kQh768EpGOqUGWvPB+1romRPiCtYgmjDQv7zgguGG9vLEHRZAf8cMn6XY9O0t10K
d+jFIIcmS9DOx99bFFNX6rEehuA5d6jbvt9OtB6hcQGgm5mNfFiuDDRlqsxVdXyo
GqRf2PbKero17bSxcy27xlTJjD+Oq0dQRxbyFv8GY9lX/uH7vmQhBF6LhUm7Hj1b
XYNXWoJvwf1QAmk1b5+E1/L1pqSqDcf1++gJ5jjJ7apVypQKcTe52vaQdqVQuS+O
Tk/ScB6F82RL/hTOs3CyyH9Oz96ZjrtBycLZJb6mg0eF24gT2cya31zzg1z/92eX
rDd477/sdYn0Q/I4tOG0LV4qSbqb30r05Ys=
In the Result Format, the Template Toolkit templating engine is used to output elements of the $serp array in a FOREACH loop.
In the result file name, you just need to change the file extension to csv.
To make the "Initial Text" option available in the Job editor, you need to activate "More options". In "Initial Text", enter the column names separated by commas, and make the second line empty.
Dump results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.query = query;
obj.links = [];
FOREACH item IN p1.serp;
obj.links.push(item.link);
END;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example result:
[{"query":"shop","links":["https://www.dns-shop.ru/","https://www.ozon.ru/","https://ru.wikipedia.org/wiki/%D0%9C%D0%B0%D0%B3%D0%B0%D0%B7%D0%B8%D0%BD","https://www.lamoda.ru/","https://www.eldorado.ru/","https://www.mvideo.ru/","https://www.wildberries.ru/","https://www.citilink.ru/","https://www.bonprix.ru/","https://www.housebrand.com/ru/ru/"]}]
To make the "Initial Text" and "Final Text" options available in the Job editor, you need to activate "More options".
Results processing
A-Parser allows you to process results directly during scraping; in this section, we have provided the most popular use cases for the Rank::Ahrefs::KeywordDifficulty scraper
Link deduplication
Add deduplication and select $serp.$i.link - Link from the dropdown list.
Download example
How to import the preset into A-Parser
eJyFVG1v0zAQ/i+nSmNSqToGAuVbVygCyjq69lNbVV5y6Uwd29hOtyjKf+ec9w4Q
Xxzf+Z7nnvOdk4Nj9mjvDFp0FoJNDrrcQwCfnlmiBe4njwZj+w2zjzyOeZgKl+3X
kv9KEYagmbFoPHIDSyaPQVCFBwEBnpSJOhBFRxgz2sIwB5dppCTqhMbwyFPxiOy1
42K/xCnTLnxkb/a1miGcmEg9YIkh03T02rAHRd+MQfFvulCl0pmshzdpPz5t6qA0
XliwASpHk0NweYTdsI5YVeHWGS4P0Hg/C/XABATOpFjsdg2LnSmTMH+FA3018nyj
uPS8uhh42u1WXly2Oe09O+FKUXTMRSfFzsi6ZYlPO4iYQ3/a8FyO3LO/FBZF3HEl
mahS+j50MuomBSAVxdLWcLQzoxJyOSwJvDNr5G5gUNpd2T8qDAQxExaHYEnqjJGQ
6OUJd2iYU2ahvR7y56DkRIg5nlB0YSX/TcpFREMziQn0pQb+PWTxB0fRltdPRW1/
MqShZSmtm8X3DhWpuTo0lyF4wh3ZduoHhLxjch4RdXtntz4sUQbbNDVznZ2ejEbp
Z6xr2UR3rrMyztpy7gyVjPlhUc9tE5nKFb3LhZwq/wR9XTIVgtpicdmNx8TWbfBG
J/AleFqmIFnt8wOnlLBf7yup2nAav3deYEI32c9aU4ZMiPVy3j+BbqTI2Kbjt9dh
uY7L9bq3f1+uH8o1As/m8KBozqjWYtf+Q9qfUP7fP0mQF9TRn/auQvryPY58dI+W
2gXBVfEbj260Iw==
See also: Results Deduplication
Link deduplication by domain
Add deduplication and select $serp.$i.link - Link from the dropdown list. Select deduplication type: Domain.
Download example
How to import the preset into A-Parser
eJyFVFtv0zAU/i9WpTGpVB0DgfLWdRQBY9269qmtKi8+6Uwd29hOtyjKf+fYuXaA
eHF8Lt93rk5BHLUHe2fAgrMkWhdEhzuJyOcXmmoBu8mTgcR+h/yaJwmPM+Hy3Ury
XxnsrlVKuSRDoqmxYDzBmiyoPERRhYoixD0rwzosejNIKF7JsCAu14Cx1BGM4QzQ
yBnKK8fFbgFTql38RN/t6qSG5EhF5gELiKlG01tDHxV+c0rKf9PFKpPO5D28yfr+
WSgHzRjGJxatCZajUSG4PJDtsPZYVu6sKbvSfhHqkQoSOZNBud02LHamTEp9Jwf6
YuT5RknQvDkbeNrNRp6dtzHtAz3CUqF3wkWXip2hdEtTH3bAqANvbXjOR+7FN4Uy
xh1XkooqpJ9Dl0Y1K8RLhb54NRzszKgUVQ4CgVfmTbprMghyV/Z9hSFRQoWFIbGY
6oxiIuy1hTsw1Ckz1z4f1BdEyYkQN3AE0bkF/quMC4ZLM0kQ9LUG/t1l/gdH2ZbX
D4VjfzaYQ8sSpKv5jw7F1I3aN80QPOUOZTv1C4LaMSoPALrt2a13S5WBNkzNXEfH
l6NB+h3rRjbRneqkjJOxnCpjJRO+n9d723hmconPcy6nyr9EX5fMhMCxWFh06zGx
9Ri80CX4GjwNIfz6Ns+POKWE/fZQpaoNx/X74BNMsZP9qDVlTIVYLW76FtKtFAqb
bPz+Mg7nOJyXvfvHcH4KJyOezcFe4Z5hreW2/Ye0/6Liv3+SqChxoj/tXYX05Xsc
6rCPFsdFoovyN1c+trs=
See also: Results Deduplication
Extracting domains
Add the Result Builder and select the source from the dropdown list: $p1.serp.$i.link - Link. Select type: Extract Domain.
Download example
How to import the preset into A-Parser
eJyFVG1v2jAQ/i8npK4SQ3TdtCnfKB3Stq60tHyCCt2SC/VwbM92aKOI/76zCQnt
Nu2L43t5nnt1avDoNu7GkiPvIFnUYOIdEvj8jIWRtBo9WsrdN6ouRZ6LtJS+Ws2V
+FXS6lIXKBT0waB1ZAPBAmaoNkmyRyUJ4560zTose2eUI1+hX4OvDHEsvSVrRUZs
FBnLcy/kakZjND59xHerJqk+bFGWATCjFA2b3lr8oflbIez+TZfqUnlbHeFtCbuH
hz4wL2fiJtoWGIrumbMBV2IGedS8OelJoTbLpTo5hdb7Drd0r9k7F5I69YSlaywC
fS9DT8F64Dkd+OeQP2aZ8EIrlPuQoWVdGvu2Ml5p9uWrFeQmVhes8hQJgrI6pLuA
XpSBKcqIvd1jIMlROuqD41QnyIlkry3Ck0Wv7dSEfFhfg1YjKa9oS7Jzi/wXpZAZ
z3eUM+hLA/y7y/QPjl1b3nEontCT5RxalihdTL93qExf6fWhGVIUwrPsxmGWrB2y
ckNk2p5dB7dCW2rDNMxNdF5yQyqsQzeykelUL8p4MZYjZQ1OlzbleIthfwFhVWJy
agNhnZr941FZTH37PNBarFjfuPuwOxGzY0yqVS7W02ZlD5FLdc8vc6rGOjzC0CdV
SsljdjTr1m3kmrEGoSv4NXgcQ3DQ9uVxDlq6r3f70o0VnNKHUHDBkzmO2lCmKOV8
dnVsgW5FWViWw/fnaTyH8Tw/un+M56d4ZhDYPK017y3XGlrQ/D7a31D9359IUu94
Q366mz0ylB9wrOM+Oh4/JGe736N7s6Y=
See also: Result Builder
Removing tags and decoding entities for anchors
Add the Result Builder and select the source from the dropdown list: $p1.serp.$i.anchor - Anchor. Select type: Remove HTML tags.
Add the Result Builder again and select the source from the dropdown list: $p1.serp.$i.anchor - Anchor. Select type: Decode HTML entities.
Download example
How to import the preset into A-Parser
eJyNVG1P2zAQ/i+nSgwpQ2Vs2pRvpazaC6OswKeCKi+5gIdje7ZTiKL+9925blPY
pvHFsc/3PPfi59JBEP7enzv0GDzk8w5s3EMOHx9FbRUuRncOK/8V2xNZVbJoVGgX
M6zNEhcnWJgSIQMrnEfHBHOYCX2f52tUnhPuwbiyx5J3iZWgLWQdhNYixSIy52Sk
kiWdr4JUFGQsbCjuxJtFSiqDpVANA2ZYCEtXr534YejbClj9m64wjQ6u3cG7BlY3
NxkQL2XiJ8bVgose2MMDqsQeVNHyam8gdHFn3PW13tuHrf+FWOKlIf9KKuzNEzqd
iZoDDEoRkG83TPsH4ZErEGUpgzRaqHVQblqfyJWWv2KC2pAvbZ1EP3GmJlPASMDG
dpPwHAbxDETRROz3NQbySiiPGXhKdSIokfL5jQzoRDBuajkfsndg9EipU1yi6t0i
/3EjVUkvPKoI9DkB/+4y/YNjtS1vNxS90YOjHLYs8XQ8/dajSnNqbjfNULKWgc5+
zK9J1iEZ7xHttmdn7FYbh9swiTlFJ5lb1CyI/slGtjc9KePJs+wYO/CmcQXFmw+z
ObBY+FWjSoAllTTo4oh8CrXia+dES8bkHVg6CcKyfRljGaftBYwEKYyu5O00zcGm
mEZf0rhP9djwZHPrdaMUKcfjrFfwyCel8KHv4XPwOIaIeaVxpiyM8l8u1t20TlJS
77LUid2oibIQSl3NTndvoFc9Ha6b4dujIq7DuB7t7N/H9UNcS2C2gLeGRoFq5Rak
f9L239b998+UdysS3U9/vkZy+YwjG/XRk6IgP1z9BrMuz2Q=
The Result Builder can be added as many times as you need.
See also: Result Builder
Filtering links by inclusion
Add a filter and select from the dropdown list: $serp.$i.link - Link. Select type: Contains string. Next, in String, specify the filtering criterion.
Download example
How to import the preset into A-Parser
eJyFVMFu2zAM/ZVACNAVyIJ1wC6+pdkCbMuaLk1OaRCoNp1qkSVNktMahv+9pOzY
TtthJ4sU+fhIPrlknruDu7XgwDsWbUpmwplF7Nszz4yE3eTRQup+QvFVpKmIc+mL
3UxID1ao/eChGDgfTkINpFAHNmKGWweW4DZsydUhimqMKEKUJ22TDgmjE0g5Htmo
ZL4wgJX1EawVCeClSNBeeyF3S5hy4+NH/nnXUByxI5c5JSwh5gavPlr+oPFbcFb9
Gy7WufK26OXbvB+fhubwGssQsWjDsB2DjtDfdtRErOrwWCvPherBjWOdoamNF1qh
7UA5Vm23J0Q30zbjNOOhuRoT9jgNng8XQypxf68uLtv67o4fYaVrYtC5cQdwwzMq
OEy4B7o94VyO/TMNiCeJIBJc1iVpJx2NtRJ/A2GlMRaPVoCbWSQfMQ8BgJzFie6G
DYNNI8hD7u86h0Uplw5GzCHVGUciyesbgQPjXttFGAr6S6bVRMo5HEF2YQH/Ohcy
QQFNUkz63iS+H7J4g1G17fVLoQSeLHJoUYJ1vfjVZSV6rvenYUiRCY+2m5JY0PsJ
nQcA087shsIybaEt0yA31fFNGVCkt25lE9O5zto4W8u5E+WViv2i0fApMlcrfLgL
NdX0RqkvlUuJa3Gw7OQxcc0ayOgIvk6ehhJIq32KzGst3Y+7mqqxAuX3hQhmOMl+
1QYy5lKul/P+DeskFeTkCDZGne41Kgi7qLbtn6L9/5T//V9EZYW7+uNu60xqjPLQ
hxNy4b1dVS8vyrR/
See also: Results Filters
Possible settings
| Parameter | Default Value | Description |
|---|---|---|
| Util::Turnstile preset | default | Select the Util::Turnstile preset for captcha bypassing. You must configure the Util::Turnstile scraper beforehand — specify your access key and other parameters, and then select the created preset here. |
| Turnstile pass proxy | ☐ | Pass proxy to the solving service. Details. |
| Country | United states | Select search country |
| Do not search for sitekey | ☑ | Experimental option; disables the search for sitekey captcha, which in turn speeds up job startup. In case of problems like an invalid sitekey, this option should be disabled. |
| Additional headers | Ability to specify arbitrary request headers |