Rank::KeysSo - Keys.so Scraper

Overview of the scraper

Keys.so — is a competitive intelligence tool. The service allows you to orient yourself with a list of competitors of your site, see which queries they rank for in search, and which queries their ads are displayed for. The largest query database in Runet is available, with over 80 million keywords.

A-Parser's functionality allows you to save scraping settings for later use (presets), schedule scraping, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results.

Result saving is possible in the form and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL and CSV.

Go to DEMO Buy A-Parser Pro ($299)

Collected Data

Visibility rating in the Russian net (Runet)
Key coverage rating
Number of pages in the TOP-50
Search traffic (Estimated daily number of users from organic search)
Number of pages in search results
Traffic per page
Number of queries per page
Performance assessment (Percentage of keys from TOP50 that rank in TOP5)
Number of sites owned by the domain owner in Adsense, Analytics, Leadia, and Relap.io
Number of subdomains of the domain owner
Number of queries in top-1, top-3, top-5, top-10, top-50
Pages in organic search results (receives a link to the page and the number of keys)
List of competitors (domain link, number of common keys, degree of domain similarity, Yandex traffic)
Key phrases (query, frequency, position)

For sites with contextual advertising:

Context traffic (Estimated traffic from context without considering any targeting. Product of exact frequency, CTR forecast, divided by 30 days)
Budget estimation (Estimated budget without considering any targeting. Product of exact frequency, CTR forecast, click cost)
Number of ads in context
Number of queries in context
Number of queries per ad
Ads in context (Ad and number of queries)
Competitors in context (Domain, number of common queries, total queries, number of ads)
Website queries in context (Query, frequency, position, placement, click cost)

Capabilities

Selection of the search result region for Google and Yandex
Scraping with authorization (by substituting cookies)

Use Cases

Getting the visibility rating in Runet, by key coverage
Getting the amount of traffic per page
Getting the number of queries in top-1, top-3, top-5, top-10, top-50
Finding website competitors
Getting a list of queries for which the site ranks in search
Getting contextual advertising ads for the website
Getting competitors in context
Collecting data on regional search results for Google and Yandex

Queries

The domain must be specified as queries, for example:

a-parser.com

Output Results Examples

A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured form, such as CSV or JSON

Default output

Result format:

Query - $query, By vision in runet - $by_vis, By keys - $by_keys, Requests in top 50 - $in_top50,  Search traf - $search_traff\n

Example result:

Query - a-parser.com, By vision in runet - 104606, By keys - 197798, Requests in top 50 - 1540,  Search traf - 57

Outputting a list of competitors

Result format:

$comp_keys.format('$links, $total_keys, $similarity, $traff_y\n')

Example result:

http://youtube.com, 1360, 0.00, 8286326
http://habr.com, 1174, 0.04, 116283
http://vc.ru, 1172, 0.06, 40749
http://zen.yandex.ru, 923, 0.00, 5944772
http://spark.ru, 870, 0.33, 2062
http://zennolab.com, 828, 2.78, 216
http://web-data-extractor.net, 819, 35.11, 53

Outputting a list of competitors in CSV

Result format:

[% FOREACH comp_keys;
  tools.CSVline(links, total_keys, similarity, traff_y);
END %]

Example result:

http://vk.com,5316893,12.18,8682202
http://youla.ru,5222139,84.47,231703
http://moskva.tiu.ru,4812708,59.84,323103
http://youtube.com,4543041,8.27,8286326
http://moskva.regmarkets.ru,4106292,61.62,13983
http://ozon.ru,3783344,46.81,580266

Dumping a list of competitors to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.competitors = [];

FOREACH item IN p1.comp_keys;
    obj.competitors.push({
        links = item.links
        total_keys = item.total_keys
        similarity = item.similarity
        traff_y = item.traff_y
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example result:

[
    {
        "query": "avito.ru",
        "competitors": [
            {
                "traffic": "8682202",
                "similarity": "12.18",
                "total_keys": "5316893",
                "links": "http://vk.com"
            },
            {
                "traffic": "231703",
                "similarity": "84.47",
                "total_keys": "5222139",
                "links": "http://youla.ru"
            },
            {
                "traffic": "323103",
                "similarity": "59.84",
                "total_keys": "4812708",
                "links": "http://moskva.tiu.ru"
            }
        ]
    }
]

tip

For the "Start text" and "End text" options to be available in the Task Editor, , you need to enable "More options".

Outputting a list of key phrases

Result format:

$key_phrases.format('$req, $base_freq, $va_freq, $pos\n')

Example result:

a parser, 2870, 332, 1
a parser download torrent, 183, 181, 1
aparser, 435, 118, 1
a parser torrent, 285, 103, 1
a parser, 423, 101, 1
a parser com, 221, 87, 1
google search results scraper, 66, 61, 1
a parser mass position checker, 53, 52, 1

Outputting a list of key phrases in CSV

Result format:

[% FOREACH key_phrases;
  tools.CSVline(req, base_freq, va_freq, pos);
END %]

Example result:

"avito",59415062,9034027,1
"avito moscow",1782404,451582,1
"avito saint petersburg",634665,343183,1
avito,1207134,307012,1
"avito voronezh",629468,286035,1

Dumping a list of key phrases to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.phrases = [];

FOREACH item IN p1.key_phrases;
    obj.phrases.push({
        req = item.req
        base_freq = item.base_freq
        va_freq = item.va_freq
        pos = item.pos
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example result:

[
    {
        "query": "avito.ru",
        "phrases": [
            {
                "base_freq": "59415062",
                "req": "avito",
                "va_freq": "9034027",
                "pos": "1"
            },
            {
                "base_freq": "1782404",
                "req": "avito moscow",
                "va_freq": "451582",
                "pos": "1"
            },
            {
                "base_freq": "634665",
                "req": "avito saint petersburg",
                "va_freq": "343183",
                "pos": "1"
            }
        ]
    }
]

tip

For the "Start text" and "End text" options to be available in the Task Editor, , you need to enable "More options".

Outputting organic search results pages

Result format:

$organic_pages.format('$pages, $keys\n')

Example result:

http://a-parser.com/, 315
http://a-parser.com/wiki/parsers/, 115
http://a-parser.com/a-parser-for-e-commerce/, 56
http://a-parser.com/wiki/js-parsers/, 44
http://a-parser.com/wiki/html-emailextractor/, 40
http://a-parser.com/resources/21/, 39
http://a-parser.com/threads/1809/, 35
http://a-parser.com/threads/4253/, 32

Outputting organic search results pages in CSV

Result format:

[% FOREACH organic_page;
    tools.CSVline(pages, keys);
END %]

Example result:

http://a-parser.com/, 315
http://a-parser.com/wiki/parsers/, 115
http://a-parser.com/a-parser-for-e-commerce/, 56
http://a-parser.com/wiki/js-parsers/, 44
http://a-parser.com/wiki/html-emailextractor/, 40
http://a-parser.com/resources/21/, 39
http://a-parser.com/threads/1809/, 35
http://a-parser.com/threads/4253/, 32

Dumping organic search results pages to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.organic = [];

FOREACH item IN p1.organic_pages;
    obj.organic.push({
        pages = item.pages
        keys = item.keys
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example result:

[
    {
        "query": "avito.ru",
        "organic": [
            {
                "keys": "4652",
                "pages": "http://avito.ru/"
            },
            {
                "keys": "4563",
                "pages": "http://avito.ru/moskva/avtomobili"
            },
            {
                "keys": "4484",
                "pages": "http://avito.ru/moskva/kvartiry/sdam-ASgBAgICAUSSA8gQ"
            }
        ]
    }
]

tip

For the "Start text" and "End text" options to be available in the Task Editor, , you need to enable "More options".

Outputting ads in context

Result format:

$ads_context.format('$aheader, $atext, $acount\n')

Example result:

PC and Laptop Repair in Moscow, On-Site! On Avito since 2015 – Bronze Status. Applications accepted 24/7! · Computers & Laptops. Tablets & Smartphones. Routers & WI - FI, 38658
Book a room on Avito. Traveling with Avito, Hotels and hotels - many offers especially for you. Now on Avito., 20122
Find a job in Moscow. On Avito Job, Fresh vacancies in Moscow. Avito - ad service #1 in Russia!, 20110

Outputting ads in context in CSV

Result format:

[% FOREACH ads_context;
  tools.CSVline(aheader, atext, acount);
END %]

Example result:

PC and Laptop Repair in Moscow","On-Site! On Avito since 2015 – Bronze Status. Applications accepted 24/7! · Computers & Laptops. Tablets & Smartphones. Routers & WI - FI",38658
"Book a room on Avito. Traveling with Avito","Hotels and hotels - many offers especially for you. Now on Avito.",20122
"Find a job in Moscow. On Avito Job","Fresh vacancies in Moscow. Avito - ad service №1 in Russia!",20110
"Dogs on Avito. Buy or take for free.","Hundreds of thousands of ads. Make beneficial deals across Russia with Avito!",17306

Dumping ads in context to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.ads = [];

FOREACH item IN p1.ads_context;
    obj.ads.push({
        header = item.aheader
        text = item.atext
        count = item.acount
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example result:

[
    {
        "ads": [
            {
                "count": "38658",
                "text": "On-Site! On Avito since 2015 – Bronze Status. Applications accepted 24/7! · Computers & Laptops. Tablets & Smartphones. Routers & WI - FI",
                "header": "PC and Laptop Repair in Moscow"
            },
            {
                "count": "20122",
                "text": "Hotels and hotels - many offers especially for you. Now on Avito.",
                "header": "Book a room on Avito. Traveling with Avito"
            },
            {
                "count": "20110",
                "text": "Fresh vacancies in Moscow. Avito - ad service №1 in Russia!",
                "header": "Find a job in Moscow. On Avito Job"
            }
        ],
        "query": "avito.ru"
    }
]

tip

For the "Start text" and "End text" options to be available in the Task Editor, , you need to enable "More options".

Outputting competitors in context

Result format:

$comp_context.format('$link, $common, $req, $ads\n')

Example result:

http://uslugi.yandex.ru, 340399, 11188812, 30632
http://superjob.ru, 180179, 885168, 19065
http://mvideo.ru, 174070, 2840150, 344731
http://pleer.ru, 166815, 2577993, 238284
http://ru.jobsora.com, 124103, 596184, 99781
http://citilink.ru, 114529, 1903350, 94937
http://goods.ru, 113373, 2684645, 328750
http://pokupki.market.yandex.ru, 81527, 2657498, 326255

Outputting competitors in context in CSV

Result format:

[% FOREACH comp_context; 
    tools.CSVline(link, common, req, ads);
END %]

Example result:

http://uslugi.yandex.ru, 340399, 11188812, 30632
http://superjob.ru, 180179, 885168, 19065
http://mvideo.ru, 174070, 2840150, 344731
http://pleer.ru, 166815, 2577993, 238284
http://ru.jobsora.com, 124103, 596184, 99781
http://citilink.ru, 114529, 1903350, 94937
http://goods.ru, 113373, 2684645, 328750
http://pokupki.market.yandex.ru, 81527, 2657498, 326255

Dumping competitors in context to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.competitors = [];

FOREACH item IN p1.comp_context;
    obj.competitors.push({
        link = item.link
        common = item.common
        req = item.req
        ads = item.ads
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example result:

[
    {
        "query": "avito.ru",
        "competitors": [
            {
                "ads": "30632",
                "link": "http://uslugi.yandex.ru",
                "req": "11188812",
                "common": "340399"
            },
            {
                "ads": "19065",
                "link": "http://superjob.ru",
                "req": "885168",
                "common": "180179"
            },
            {
                "ads": "344731",
                "link": "http://mvideo.ru",
                "req": "2840150",
                "common": "174070"
            },
            
        ]
    }
]

tip

For the "Start text" and "End text" options to be available in the Task Editor, , you need to enable "More options".

Outputting website queries in context

Result format:

$wreq_context.format('$request, $freq, $va_freq, $pos, $placement, $price\n')

Example result:

job in chelyabinsk, 202530, 40163, 3, Spec., 4
algebra 8th grade dorofeev, 217307, 18467, 2, Gar., 2
far cry 3, 113266, 23218, 5, Gar., 3
job in barnaul, 128775, 26402, 3, Spec., 4
harry potter and the chamber of secrets movie 2002, 36482, 30450, 2, Gar., 5
mazda 6, 554463, 25248, 6, Gar., 19
assassin s creed, 535915, 29419, 5, Gar., 5
avito ryazan, 325994, 137938, 3, Gar., 13

Outputting website queries in context in CSV

Result format:

[% FOREACH wreq_context;
    tools.CSVline(request, freq, va_freq, pos, placement, price);
END %]

Example result:

"job in chelyabinsk",202530,40163,3,"Spec.",4
"algebra 8th grade dorofeev",217307,18467,2,"Gar.",2
"far cry 3",113266,23218,5,"Gar.",3
"job in barnaul",128775,26402,3,"Spec.",4
"harry potter and the chamber of secrets movie 2002",36482,30450,2,"Gar.",5
"mazda 6",554463,25248,6,"Gar.",19
"assassin s creed",535915,29419,5,"Gar.",5
"avito ryazan",325994,137938,3,"Gar.",13
"avito ryazan region",72077,19586,2,"Gar.",20

Dumping website queries in context to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.wreq = [];

FOREACH item IN p1.wreq_context;
    obj.wreq.push({
        request = item.request
        freq = item.freq
        va_freq = item.va_freq
        pos = item.pos
        placement = item.placement
        price = item.price
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example result:

[
    {
        "query": "avito.ru",
        "wreq": [
            {
                "request": "job in chelyabinsk",
                "placement": "Spec.",
                "freq": "202530",
                "va_freq": "40163",
                "price": "4",
                "pos": "3"
            },
            {
                "request": "algebra 8th grade dorofeev",
                "placement": "Gar.",
                "freq": "217307",
                "va_freq": "18467",
                "price": "2",
                "pos": "2"
            },
            {
                "request": "far cry 3",
                "placement": "Gar.",
                "freq": "113266",
                "va_freq": "23218",
                "price": "3",
                "pos": "5"
            }
        ]
    }
]

tip

For the "Start text" and "End text" options to be available in the Task Editor, , you need to enable "More options".

Possible Settings

note

Common settings for all scrapers

Parameter	Default value	Description
Search DB	`Yandex Moscow`	Search base selection
Cookie "userlogin="		In this field, you must specify the value of the `userlogin` cookie taken from the browser after authorization

Overview of the scraper​

Collected Data​

Capabilities​

Use Cases​

Queries​

Output Results Examples​

Default output​

Outputting a list of competitors​

Outputting a list of competitors in CSV​

Dumping a list of competitors to JSON​

Outputting a list of key phrases​

Outputting a list of key phrases in CSV​

Dumping a list of key phrases to JSON​

Outputting organic search results pages​

Outputting organic search results pages in CSV​

Dumping organic search results pages to JSON​

Outputting ads in context​

Outputting ads in context in CSV​

Dumping ads in context to JSON​

Outputting competitors in context​

Outputting competitors in context in CSV​

Dumping competitors in context to JSON​

Outputting website queries in context​

Outputting website queries in context in CSV​

Dumping website queries in context to JSON​

Possible Settings​

Overview of the scraper

Collected Data

Capabilities

Use Cases

Queries

Output Results Examples

Default output

Outputting a list of competitors

Outputting a list of competitors in CSV

Dumping a list of competitors to JSON

Outputting a list of key phrases

Outputting a list of key phrases in CSV

Dumping a list of key phrases to JSON

Outputting organic search results pages

Outputting organic search results pages in CSV

Dumping organic search results pages to JSON

Outputting ads in context

Outputting ads in context in CSV

Dumping ads in context to JSON

Outputting competitors in context

Outputting competitors in context in CSV

Dumping competitors in context to JSON

Outputting website queries in context

Outputting website queries in context in CSV

Dumping website queries in context to JSON

Possible Settings