Rank::KeysSo - Keys.so scraper

Scraper Overview

Keys.so is a competitive intelligence tool. The service allows you to navigate through the list of competitors of your site, see which queries they rank for in search, and which queries their ads are displayed for. The largest database of queries in the Runet is available, more than 80 million keywords.

A-Parser functionality allows you to save parsing settings for future use (presets), set a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, permutation of alphanumeric combinations and lists to get the maximum possible number of results.

Saving results is possible in the form and structure that you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Go to DEMO Buy A-Parser Pro ($299)

Collected Data

Rating by visibility in the Runet
Rating by keyword coverage
Number of pages in the TOP-50
Search traffic (Approximate number of users from organic search per day)
Number of pages in search results
Traffic per page
Number of queries per page
Effectiveness rating (Percentage of keys from the TOP50 occupying positions in the TOP5)
Number of sites owned by the domain in Adsense, Analytics, Leadia, and Relap.io
Number of subdomains owned by the domain
Number of queries in the top-1, top-3, top-5, top-10, top-50
Pages in organic search results (gets a link to the page and the number of keys)
List of competitors (link to the domain, number of common keys, degree of domain similarity, Yandex traffic)
Key phrases (query, frequency, position)

For sites with contextual advertising:

Context traffic (Estimate of traffic from context without considering any targeting. The product of exact frequency, forecast CTR, divided by 30 days)
Budget estimate (Budget estimate without considering any targeting. The product of exact frequency, forecast CTR, cost per click)
Number of ads in context
Number of queries in context
Number of queries per ad
Ads in context (Ad and number of queries)
Competitors in context (Domain, number of common queries, total queries, number of ads)
Site's queries in context (Query, frequency, position, placement, cost per click)

Capabilities

Selection of the search region for Google and Yandex
Parsing with authorization (via cookie substitution)

Use Cases

Obtaining a rating by visibility in the Runet, by keyword coverage
Obtaining the amount of traffic per page
Obtaining the number of queries in the top-1, top-3, top-5, top-10, top-50
Searching for site competitors
Obtaining a list of queries by which the site ranks in search
Obtaining site's contextual advertising ads
Obtaining competitors in context
Collecting data by regional search for Google and Yandex

Queries

As queries, you need to specify the domain, for example:

a-parser.com

Output Results Examples

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON

Default output

Result format:

Query - $query, By vision in runet - $by_vis, By keys - $by_keys, Requests in top 50 - $in_top50,  Search traf - $search_traff\n

Example of result:

Query - a-parser.com, By vision in runet - 104606, By keys - 197798, Requests in top 50 - 1540,  Search traf - 57

Competitors list output

Result format:

$comp_keys.format('$links, $total_keys, $similarity, $traff_y\n')

Example of result:

http://youtube.com, 1360, 0.00, 8286326
http://habr.com, 1174, 0.04, 116283
http://vc.ru, 1172, 0.06, 40749
http://zen.yandex.ru, 923, 0.00, 5944772
http://spark.ru, 870, 0.33, 2062
http://zennolab.com, 828, 2.78, 216
http://web-data-extractor.net, 819, 35.11, 53

Competitors list output in CSV

Result format:

[% FOREACH comp_keys;
  tools.CSVline(links, total_keys, similarity, traff_y);
END %]

Example of result:

http://vk.com,5316893,12.18,8682202
http://youla.ru,5222139,84.47,231703
http://moskva.tiu.ru,4812708,59.84,323103
http://youtube.com,4543041,8.27,8286326
http://moskva.regmarkets.ru,4106292,61.62,13983
http://ozon.ru,3783344,46.81,580266

Dump competitors list to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.competitors = [];

FOREACH item IN p1.comp_keys;
    obj.competitors.push({
        links = item.links
        total_keys = item.total_keys
        similarity = item.similarity
        traff_y = item.traff_y
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example of result:

[
    {
        "query": "avito.ru",
        "competitors": [
            {
                "traffic": "8682202",
                "similarity": "12.18",
                "total_keys": "5316893",
                "links": "http://vk.com"
            },
            {
                "traffic": "231703",
                "similarity": "84.47",
                "total_keys": "5222139",
                "links": "http://youla.ru"
            },
            {
                "traffic": "323103",
                "similarity": "59.84",
                "total_keys": "4812708",
                "links": "http://moskva.tiu.ru"
            }
        ]
    }
]

tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Key phrases list output

Result format:

$key_phrases.format('$req, $base_freq, $va_freq, $pos\n')

Example of result:

a parser, 2870, 332, 1
a parser скачать торрент, 183, 181, 1
aparser, 435, 118, 1
a parser торрент, 285, 103, 1
а парсер, 423, 101, 1
a parser com, 221, 87, 1
парсер выдачи google, 66, 61, 1
массовый чекер позиций a parser, 53, 52, 1

Key phrases list output in CSV

Result format:

[% FOREACH key_phrases;
  tools.CSVline(req, base_freq, va_freq, pos);
END %]

Example of result:

"авито",59415062,9034027,1
"авито москва",1782404,451582,1
"авито санкт петербург",634665,343183,1
avito,1207134,307012,1
"авито воронеж",629468,286035,1

Dump key phrases list to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.phrases = [];

FOREACH item IN p1.key_phrases;
    obj.phrases.push({
        req = item.req
        base_freq = item.base_freq
        va_freq = item.va_freq
        pos = item.pos
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example of result:

[
    {
        "query": "avito.ru",
        "phrases": [
            {
                "base_freq": "59415062",
                "req": "авито",
                "va_freq": "9034027",
                "pos": "1"
            },
            {
                "base_freq": "1782404",
                "req": "авито москва",
                "va_freq": "451582",
                "pos": "1"
            },
            {
                "base_freq": "634665",
                "req": "авито санкт петербург",
                "va_freq": "343183",
                "pos": "1"
            }
        ]
    }
]

tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Organic pages output

Result format:

$organic_pages.format('$pages, $keys\n')

Example of result:

http://a-parser.com/, 315
http://a-parser.com/wiki/parsers/, 115
http://a-parser.com/a-parser-for-e-commerce/, 56
http://a-parser.com/wiki/js-parsers/, 44
http://a-parser.com/wiki/html-emailextractor/, 40
http://a-parser.com/resources/21/, 39
http://a-parser.com/threads/1809/, 35
http://a-parser.com/threads/4253/, 32

Organic pages output in CSV

Result format:

[% FOREACH organic_page;
    tools.CSVline(pages, keys);
END %]

Example of result:

http://a-parser.com/, 315
http://a-parser.com/wiki/parsers/, 115
http://a-parser.com/a-parser-for-e-commerce/, 56
http://a-parser.com/wiki/js-parsers/, 44
http://a-parser.com/wiki/html-emailextractor/, 40
http://a-parser.com/resources/21/, 39
http://a-parser.com/threads/1809/, 35
http://a-parser.com/threads/4253/, 32

Dump organic pages to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.organic = [];

FOREACH item IN p1.organic_pages;
    obj.organic.push({
        pages = item.pages
        keys = item.keys
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example of result:

[
    {
        "query": "avito.ru",
        "organic": [
            {
                "keys": "4652",
                "pages": "http://avito.ru/"
            },
            {
                "keys": "4563",
                "pages": "http://avito.ru/moskva/avtomobili"
            },
            {
                "keys": "4484",
                "pages": "http://avito.ru/moskva/kvartiry/sdam-ASgBAgICAUSSA8gQ"
            }
        ]
    }
]

tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Context ads output

Result format:

$ads_context.format('$aheader, $atext, $acount\n')

Example result:

Ремонт ПК и Ноутбуков в Москве, на Выезде! С 2015 года на Авито – Бронзовый статус. Прием заявок 24/7! · Компьютеры & Ноутбуки. Планшеты & Смартфоны. Роутеры & WI - FI, 38658
Забронируйте номер на Авито. В поездку вместе с Авито, Гостиницы и отели - множество предложений специально для вас. Теперь на Авито., 20122
Найдите работу в городе Москва. На Авито Работа, Свежие вакансии в городе Москва. Авито - сервис объявлений №1 в России!, 20110

Context ads output in CSV

Result format:

[% FOREACH ads_context;
  tools.CSVline(aheader, atext, acount);
END %]

Example result:

Ремонт ПК и Ноутбуков в Москве","на Выезде! С 2015 года на Авито – Бронзовый статус. Прием заявок 24/7! · Компьютеры & Ноутбуки. Планшеты & Смартфоны. Роутеры & WI - FI",38658
"Забронируйте номер на Авито. В поездку вместе с Авито","Гостиницы и отели - множество предложений специально для вас. Теперь на Авито.",20122
"Найдите работу в городе Москва. На Авито Работа","Свежие вакансии в городе Москва. Авито - сервис объявлений №1 в России!",20110
"Собаки на Авито. Купить или взять бесплатно.","Сотни тысяч объявлений. Совершайте выгодные сделки по всей России с помощью Авито!",17306

Dump context ads to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.ads = [];

FOREACH item IN p1.ads_context;
    obj.ads.push({
        header = item.aheader
        text = item.atext
        count = item.acount
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example of result:

[
    {
        "ads": [
            {
                "count": "38658",
                "text": "на Выезде! С 2015 года на Авито – Бронзовый статус. Прием заявок 24/7! · Компьютеры & Ноутбуки. Планшеты & Смартфоны. Роутеры & WI - FI",
                "header": "Ремонт ПК и Ноутбуков в Москве"
            },
            {
                "count": "20122",
                "text": "Гостиницы и отели - множество предложений специально для вас. Теперь на Авито.",
                "header": "Забронируйте номер на Авито. В поездку вместе с Авито"
            },
            {
                "count": "20110",
                "text": "Свежие вакансии в городе Москва. Авито - сервис объявлений №1 в России!",
                "header": "Найдите работу в городе Москва. На Авито Работа"
            }
        ],
        "query": "avito.ru"
    }
]

tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Context competitors output

Result format:

$comp_context.format('$link, $common, $req, $ads\n')

Example of result:

http://uslugi.yandex.ru, 340399, 11188812, 30632
http://superjob.ru, 180179, 885168, 19065
http://mvideo.ru, 174070, 2840150, 344731
http://pleer.ru, 166815, 2577993, 238284
http://ru.jobsora.com, 124103, 596184, 99781
http://citilink.ru, 114529, 1903350, 94937
http://goods.ru, 113373, 2684645, 328750
http://pokupki.market.yandex.ru, 81527, 2657498, 326255

Context competitors output in CSV

Result format:

[% FOREACH comp_context; 
    tools.CSVline(link, common, req, ads);
END %]

Example of result:

http://uslugi.yandex.ru, 340399, 11188812, 30632
http://superjob.ru, 180179, 885168, 19065
http://mvideo.ru, 174070, 2840150, 344731
http://pleer.ru, 166815, 2577993, 238284
http://ru.jobsora.com, 124103, 596184, 99781
http://citilink.ru, 114529, 1903350, 94937
http://goods.ru, 113373, 2684645, 328750
http://pokupki.market.yandex.ru, 81527, 2657498, 326255

Dump context competitors to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.competitors = [];

FOREACH item IN p1.comp_context;
    obj.competitors.push({
        link = item.link
        common = item.common
        req = item.req
        ads = item.ads
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example of result:

[
    {
        "query": "avito.ru",
        "competitors": [
            {
                "ads": "30632",
                "link": "http://uslugi.yandex.ru",
                "req": "11188812",
                "common": "340399"
            },
            {
                "ads": "19065",
                "link": "http://superjob.ru",
                "req": "885168",
                "common": "180179"
            },
            {
                "ads": "344731",
                "link": "http://mvideo.ru",
                "req": "2840150",
                "common": "174070"
            },
            
        ]
    }
]

tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Context site queries output

Result format:

$wreq_context.format('$request, $freq, $va_freq, $pos, $placement, $price\n')

Example of result:

работа в челябинске, 202530, 40163, 3, Спец., 4
алгебра 8 класс дорофеев, 217307, 18467, 2, Гар., 2
far cry 3, 113266, 23218, 5, Гар., 3
работа в барнауле, 128775, 26402, 3, Спец., 4
гарри поттер и тайная комната фильм 2002, 36482, 30450, 2, Гар., 5
мазда 6, 554463, 25248, 6, Гар., 19
assassin s creed, 535915, 29419, 5, Гар., 5
авито рязань, 325994, 137938, 3, Гар., 13

Context site queries output in CSV

Result format:

[% FOREACH wreq_context;
    tools.CSVline(request, freq, va_freq, pos, placement, price);
END %]

Example of result:

"работа в челябинске",202530,40163,3,"Спец.",4
"алгебра 8 класс дорофеев",217307,18467,2,"Гар.",2
"far cry 3",113266,23218,5,"Гар.",3
"работа в барнауле",128775,26402,3,"Спец.",4
"гарри поттер и тайная комната фильм 2002",36482,30450,2,"Гар.",5
"мазда 6",554463,25248,6,"Гар.",19
"assassin s creed",535915,29419,5,"Гар.",5
"авито рязань",325994,137938,3,"Гар.",13
"авито рязанская область",72077,19586,2,"Гар.",20

Dump context site queries to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.wreq = [];

FOREACH item IN p1.wreq_context;
    obj.wreq.push({
        request = item.request
        freq = item.freq
        va_freq = item.va_freq
        pos = item.pos
        placement = item.placement
        price = item.price
    });
END;

obj.json %]

Начальный текст:

Конечный текст:

Example of result:

[
    {
        "query": "avito.ru",
        "wreq": [
            {
                "request": "работа в челябинске",
                "placement": "Спец.",
                "freq": "202530",
                "va_freq": "40163",
                "price": "4",
                "pos": "3"
            },
            {
                "request": "алгебра 8 класс дорофеев",
                "placement": "Гар.",
                "freq": "217307",
                "va_freq": "18467",
                "price": "2",
                "pos": "2"
            },
            {
                "request": "far cry 3",
                "placement": "Гар.",
                "freq": "113266",
                "va_freq": "23218",
                "price": "3",
                "pos": "5"
            }
        ]
    }
]

tip

To make the "Initial text" and "Final text" options available in the Task Editor, you need to activate "More options".

Possible Settings

note

Common settings for all scrapers

Parameter	Default Value	Description
Search DB	`Yandex Moscow`	Choice of search database
Cookie "userlogin="		In this field, you need to specify the value of the cookie `userlogin`, taken from the browser after authorization

Scraper Overview​

Collected Data​

Capabilities​

Use Cases​

Queries​

Output Results Examples​

Default output​

Competitors list output​

Competitors list output in CSV​

Dump competitors list to JSON​

Key phrases list output​

Key phrases list output in CSV​

Dump key phrases list to JSON​

Organic pages output​

Organic pages output in CSV​

Dump organic pages to JSON​

Context ads output​

Context ads output in CSV​

Dump context ads to JSON​

Context competitors output​

Context competitors output in CSV​

Dump context competitors to JSON​

Context site queries output​

Context site queries output in CSV​

Dump context site queries to JSON​

Possible Settings​

Scraper Overview

Collected Data

Capabilities

Use Cases

Queries

Output Results Examples

Default output

Competitors list output

Competitors list output in CSV

Dump competitors list to JSON

Key phrases list output

Key phrases list output in CSV

Dump key phrases list to JSON

Organic pages output

Organic pages output in CSV

Dump organic pages to JSON

Context ads output

Context ads output in CSV

Dump context ads to JSON

Context competitors output

Context competitors output in CSV

Dump context competitors to JSON

Context site queries output

Context site queries output in CSV

Dump context site queries to JSON

Possible Settings