SE::Yandex::Direct - Yandex Direct ad scraper direct.yandex.ru
Overview of the scraper
SE::Yandex::Direct – Yandex Direct ads scraper. Automated collection of all data: number of ads for a specific query, title, text, domain, additional links and their anchors, tags.You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results. Using result filtering you can immediately clean the result, removing all unnecessary garbage (using minus words).
A-Parser functionality allows you to save the parsing settings of the SE::Yandex::Direct scraper for future use (presets), set a parsing schedule and much more.
Saving the results is possible in the format and structure that you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL and CSV.
Data collected
- Number of ads for a specific query
- Title, text and domain
- Additional links and their anchors
- Tags
- Whether the result is an advertisement
Capabilities
- Parses the maximum number of results offered by Yandex Direct – 10 pages
- Supports selecting the search region
- Ability to bypass captcha using AntiCaptcha or any other service supporting their API (it is recommended to enable the is_russian parameter in the anticaptcha settings)
Use cases
- Collecting competitors advertised for specific queries
Queries
Search phrases should be specified as queries, for example:
Buy kite
Photoshop
Windows in Moscow
Apartment renovation
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add a list of other words to each query, we will specify several main queries:
fantasy
tower defense
rpg
In the query format, we will specify a macro for substituting additional words from the file keywords.txt, this method allows you to increase the variability of requests many times over:
{subs:keywords} $query
This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro operation.
For example, if the file keywords.txt contains:
free
online
As a result, the substitution macro will turn 3 main queries into 6:
free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg
Output results examples
A-Parser supports flexible results formatting thanks to the built-in template engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured form, such as CSV or JSON
Default output
Result format:
$query - $totalcount\n$ads.format('$domain: $title - $text\n')
Example result:
Photoshop - 5
www.syssoft.ru: Adobe Photoshop CS5 - Discounts - Photoshop CS5 with a discount from an authorized partner. Order now!
www.disted.ru: Learn photography! - Distance learning in Russian at the New York Institute of Photography.
www.softkey.ru: Licensed Adobe Photoshop CS! - The best new image editing features from Adobe Photoshop.
www.softprof-it.ru: Photoshop license from 23000 rubles - Low price Photoshop - from 23000 rubles. Call. Clarify purchase conditions.
photoshop.pro1904.ru: Virtuoso work in Photoshop! - Unique course "Photoshop from scratch". Become a Photoshop Master.
Buy kite - 0
Windows in Moscow - 33
www.imperia51.com: Windows in Murmansk! Low prices! - Very low prices! Guarantee! Quality installation! All this is «Imperiya Okon»!
mirperegorodok-kms.ru: «Mir Peregorodok» Company - Aluminum systems: office partitions, entrance groups, doors...
inkar-avto.ru: Moscow-St. Petersburg-Moscow by bus - Mon, Tue, Wed, Thu - 1000 rub, Fri, Sat, Sun - 1300.
Output in CSV table
General result format:
[% FOREACH i IN p1.ads;
tools.CSVline(i.domain, i.title, i.text);
END %]
File name:
$datefile.format().csv
Initial text:
Domain,Title,Text
The Template Toolkit templating engine is used in the General Result Format to output elements of the $ads array in a FOREACH loop.
In the result file name, you just need to change the file extension to csv.
For the "Initial text" option to be available in the Task Editor, you need to activate "More options". In the "Initial text" field, write the column names separated by commas and make the second line empty.
Saving in SQL Format
Result format:
[% FOREACH ads;
"INSERT INTO ads VALUES('" _ domain _ "', '" _ title _ "', '" _ text _ "')\n";
END %]
Result example:
INSERT INTO ads VALUES('oknamr.ru', 'Plastic <b>windows</b> cheap <b>in</b> <b>Moscow</b>: купить по низким...', 'Компания «<b>WINDOWS</b> <b>MOSCOW</b>» installs plastic <b>windows</b>, doors and partitions, as well as glazing balconies <b>in</b> <b>Moscow</b> and Moscow region.')
INSERT INTO ads VALUES('gor-okna.ru', 'Buy <b>windows</b> PVC cheaply | <b>Windows</b> PVC <b>in</b> <b>Moscow</b> по ценам...', 'Заказать качественные <b>windows</b> PVC <b>in</b> <b>Moscow</b> with delivery and installation. ... <b>Moscow</b>. from 9:00 to 21:00 daily.')
INSERT INTO ads VALUES('balcon-msk.ru', 'Glazing of balconies and loggias <b>in</b> <b>Moscow</b>.', 'Company «Balconies <b>Moscow</b>» produces professional glazing of balconies and loggias ... The price includes: <b>Windows</b> aluminum 3.2x0.7m h-1.5m, water run-off, installation...')
INSERT INTO ads VALUES('windowsмаркет33.рф', 'Весна! Время купить новые <b>окна</b> PVC! Gift – Mosquito…', '<b>Window</b> 1.3h1.4 for 9900 turnkey in a panel house! <b>Window</b> 2.0h1.4 for 13500 turnkey!')
INSERT INTO ads VALUES('topokno72.ru', 'Plastic <b>windows</b>. Двери, балконы', 'Пластиковые <b>windows</b> and any PVC constructions from WinTech, Exprof, Proplex, Rehau, KBE')
INSERT INTO ads VALUES('okna-moskvy1.ru', '<b>Windows</b> <b>Moscow</b>. Официальный сайт', 'Только до конца месяца. <b>Windows</b> REHAU with a discount up to 60%. Multi-glass as a gift. Click!')
INSERT INTO ads VALUES('russia.superjob.ru', 'Plastic and aluminum <b>windows</b>', 'New colleagues are already waiting for you! Convenient search, free recommendations, fast import. · 20 years on the market. Recommendations. Convenient search. Online support')
INSERT INTO ads VALUES('gorokna-11.ru', 'Plastic <b>windows</b> <b>in</b> Vorkuta. <b>Windows</b>,лоджии,балконы', '<b>Windows</b> PVC with a discount in Vorkuta')
INSERT INTO ads VALUES('shop.oknagorizont.ru', 'Online store <b>windows</b>, cheap plastic <b>windows</b>...', 'Online store <b>windows</b> without installation, cheap plastic <b>windows</b> today, low prices for <b>windows</b>. Many in stock and custom, urgent manufacturing...')
INSERT INTO ads VALUES('msk-oknaidveri.ru', 'Plastic <b>windows</b> ПВХ и входные металлические двери...', 'Классические ПВХ-<b>windows</b> - are <b>windows</b>, made of plastic, certified by the Ministry of Construction of Russia and compliant with SNiP and others...')
INSERT INTO ads VALUES('oknamag.pro', 'Plastic <b>windows</b> для дачи купить в Окнамаг Про', 'Размеры пластикового <b>window</b>: 600 height, 600 width. Available options are fabri.. Quick view. Product code: opd0606.')
INSERT INTO ads VALUES('фурнитура-windows.рф', 'Фурнитура для пластиковых <b>windows</b> - Запчасти...', 'Фурнитура для <b>окон</b> <b>retail and wholesale. We will help you make the right choice of components and spare parts for your plastic </b> розницу и оптом. Поможем сделать правильный выбор комплектующих и запчастей для ваших пластиковых <b>windows</b>.')
INSERT INTO ads VALUES('OknaGarant.ru', 'Ready-made plastic <b>windows</b> <b>in</b> <b>Moscow</b>, купить новые...', '<b>Windows</b> are already waiting for you in our warehouse! The warehouse is located <b>in</b> <b>Moscow</b> within the Moscow Ring Road. Savings, the cost is significantly lower (compared to <b>windows</b> to order).')
INSERT INTO ads VALUES('domaster.ru', 'Window accessories and fittings buy <b>in</b> <b>Moscow</b>...', 'For <b>windows</b> and doors. Online store Brass — is the place where you can buy accessories for caring for <b>windows</b> and fittings from different companies with delivery across Russia.')
INSERT INTO ads VALUES('sms-m.ru', 'Plastic <b>windows</b> manufacturing <b>in</b> <b>Moscow</b>', 'When the question arose about minimizing costs in the manufacturing of modular buildings, StroiMontazhService-M decided to launch their own...')
INSERT INTO ads VALUES('rosband.ru', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b>', 'Plastic <b>windows</b> have excellent thermal insulation properties and there is no need to spend more ... Moscow, Dobryninsky lane, 19 stro. 6.')<b>Москва</b>, 1-й Добрынинский переулок, 19 стр. 6.')
INSERT INTO ads VALUES('moskva.miltor.ru', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b>: 185 предложений', 'Купить пластиковые <b>windows</b> <b>in</b> <b>Moscow</b> using the ads on our site. Here you can find out the cost of plastic <b>windows</b>, ordering conditions and address...')
INSERT INTO ads VALUES('city-jaluzi.ru', 'Blinds for <b>windows</b> buy cheaply <b>in</b> <b>Moscow</b> - отзывы, фото...', 'Производим жалюзи на <b>windows</b>, sew curtains, assemble cornices and roller shutters to order since 2006 Prices from 495 to 5 900 ₽ per 1 sq.m, Free measurement and...')
Dump Results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.ads = [];
FOREACH item IN p1.ads;
obj.ads.push({
domain = item.domain
title = item.title
text = item.text
});
END;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Result example:
[{"ads": [{"domain":"oknamr.ru","text":"Company «<b>WINDOWS</b> <b>MOSCOW</b>» installs plastic <b>windows</b>, doors and partitions, as well as glazing balconies <b>in</b> <b>Moscow</b> and Moscow region.","title":"Plastic <b>windows</b> cheap <b>in</b> <b>Moscow</b>: buy low..."},{"domain":"gor-okna.ru","text":"Order quality <b>windows</b> PVC <b>in</b> <b>Moscow</b> with delivery and installation. ... <b>Moscow</b>. from 9:00 to 21:00 daily.","title":"Buy <b>windows</b> PVC cheaply | <b>Windows</b> PVC <b>in</b> <b>Moscow</b> at prices..."},{"domain":"balcon-msk.ru","text":"Company «Balconies <b>Moscow</b>» produces professional glazing of balconies and loggias ... The price includes: <b>Windows</b> aluminum 3.2x0.7m h-1.5m, water run-off, installation...","title":"Glazing of balconies and loggias <b>in</b> <b>Moscow</b>."},{"domain":"oknamarket33.rf","text":"<b>Window</b> 1.3h1.4 for 9900 turnkey in a panel house! <b>Window</b> 2.0h1.4 for 13500 turnkey!","title":"Spring! Time to buy new <b>windows</b> PVC! Gift – Mosquito…"},{"domain":"okna-moskvy1.ru","text":"Only until the end of the month. <b>Windows</b> REHAU with a discount up to 60%. Multi-glass as a gift. Click!","title":"<b>Windows</b> <b>Moscow</b>. Official website"},{"domain":"okno61.rf","text":"Plastic <b>windows</b> at low prices! Quality installation. Fast delivery. Guarantee.","title":"<b>Windows</b> <b>in</b> Bataysk! Seasonal discounts"},{"domain":"gorokna-11.ru","text":"<b>Windows</b> PVC with a discount in Vorkuta","title":"Plastic <b>windows</b> <b>in</b> Vorkuta. <b>Windows</b>,loggias,balconies"}]}]
For the "Initial text" and "Final text" options to be available in the Task Editor, you need to activate "More options".
Possible Settings
| Parameter | Default value | Description |
|---|---|---|
| Pages count | 10 | Number of pages to scrape |
| Region of serp (lr=) | Moscow | Select search region (parameter lr=) |
| AntiGate preset | default | Using a captcha solving service. You need to pre-configure the  Util::AntiGate scraper - specify your access key and other parameters, and then select the created preset here |