SE::Yandex::Direct - Yandex Direct ad parser direct.yandex.ru
Parser overview
SE::Yandex::Direct – Yandex Direct ads parser. Automated collection of all data: number of ads for a specific query, title, text, domain, sitelinks and their anchors, tags.You can use automatic query multiplication, substitution of subqueries from files, enumeration of alphanumeric combinations and lists to obtain the maximum possible number of results. Using results filtering you can immediately clean the result by removing all unnecessary garbage (using negative keywords).
The A-Parser functionality allows you to save parsing settings for the SE::Yandex::Direct parser for future use (presets), set a parsing schedule, and much more.
Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
Collected data
- Number of ads for a specific query
- Title, text, and domain
- Sitelinks and their anchors
- Tags
- Whether the result is an advertisement
Capabilities
- Parses the maximum number of results provided by Yandex Direct – 10 pages
- Supports search region selection
- Ability to bypass captcha using the AntiCaptcha service or any other supporting their API (it is recommended to enable the is_russian parameter in the anti-gate settings)
Use cases
- Collecting competitors advertising for specified queries
Queries
Search phrases should be specified as queries, for example:
Buy kite
Photoshop
Windows in Moscow
Apartment renovation
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files; for example, if we want to add a list of other words to each query, we specify several main queries:
fantasy
tower defense
rpg
In the query format, we specify a macro for substituting additional words from the keywords.txt file; this method allows increasing query variability manifold:
{subs:keywords} $query
This macro will create as many additional queries as there are in the file for each original search query, which will result in [number of original queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro's operation.
For example, if the keywords.txt file contains:
free
online
As a result, the substitution macro will turn 3 main queries into 6:
free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg
Output results examples
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in arbitrary form, as well as structured, for example CSV or JSON
Default output
Result format:
$query - $totalcount\n$ads.format('$domain: $title - $text\n')
Result example:
Photoshop - 5
www.syssoft.ru: Adobe Photoshop CS5 - Discounts - Photoshop CS5 with a discount from an authorized partner. Order now!
www.disted.ru: Learn to take photos! - Distance learning in Russian at the New York Institute of Photography.
www.softkey.ru: Licensed Adobe Photoshop CS! - Best new image editing features from Adobe Photoshop.
www.softprof-it.ru: Photoshop license from 23,000 rubles - Low price for Photoshop - from 23,000 rubles. Call. Check purchase terms.
photoshop.pro1904.ru: Masterful work in Photoshop! - Unique course "Photoshop from scratch". Become a Photoshop Master.
Buy kite - 0
Windows in Moscow - 33
www.imperia51.com: Windows in Murmansk! Low prices! - Very low prices! Warranty! Quality installation! All this is «Empire of Windows»!
mirperegorodok-kms.ru: Company «Mir Peregorodok» - Aluminum systems: office partitions, entrance groups, doors...
inkar-avto.ru: Moscow-St. Petersburg-Moscow by bus - Mon, Tue, Wed, Thu - 1000 rub, Fri, Sat, Sun - 1300.
Output in CSV table
General result format:
[% FOREACH i IN p1.ads;
tools.CSVline(i.domain, i.title, i.text);
END %]
File name:
$datefile.format().csv
Initial text:
Domain,Title,Text
In the General result format, the Template Toolkit template engine is used to output elements of the $ads array in a FOREACH loop.
In the result file name, you just need to change the file extension to csv.
To make the "Initial text" option available in the Task Editor, you need to activate "More options". In "Initial text", write the column names separated by commas and make the second line empty.
Saving in SQL format
Result format:
[% FOREACH ads;
"INSERT INTO ads VALUES('" _ domain _ "', '" _ title _ "', '" _ text _ "')\n";
END %]
Result example:
INSERT INTO ads VALUES('oknamr.ru', 'Plastic <b>windows</b> inexpensive <b>in</b> <b>Moscow</b>: buy at low...', 'The company «<b>WINDOWS</b> <b>OF MOSCOW</b>» performs installation of plastic <b>windows</b>, doors and partitions, as well as glazing of balconies <b>in</b> <b>Moscow</b> and Moscow region.')
INSERT INTO ads VALUES('gor-okna.ru', 'Buy <b>windows</b> PVC inexpensive | <b>Windows</b> PVC <b>in</b> <b>Moscow</b> at prices...', 'Order high-quality <b>windows</b> PVC <b>in</b> <b>Moscow</b> with delivery and installation. ... <b>Moscow</b>. from 9:00 to 21:00 daily.')
INSERT INTO ads VALUES('balcon-msk.ru', 'Glazing of balconies and loggias <b>in</b> <b>Moscow</b>.', 'The company "Balconies of <b>Moscow</b>" performs professional glazing of balconies and loggias ... The price includes: aluminum <b>Windows</b> 3.2x0.7m h-1.5m, water drainage, installation...')
INSERT INTO ads VALUES('oknamarket33.rf', 'Spring! Time to buy new PVC <b>windows</b>! Gift – Mosquito...', '<b>Window</b> 1.3x1.4 for 9900 turnkey in a panel house! <b>Window</b> 2.0x1.4 for 13500 turnkey!')
INSERT INTO ads VALUES('topokno72.ru', 'Plastic <b>windows</b>. Doors, balconies', 'Plastic <b>windows</b> and any structures made of PVC WinTech, Exprof, Proplex, Rehau, KBE')
INSERT INTO ads VALUES('okna-moskvy1.ru', '<b>Windows</b> <b>of Moscow</b>. Official site', 'Only until the end of the month. <b>Windows</b> REHAU with a discount up to 60%. Multi-glass as a gift. Click!')
INSERT INTO ads VALUES('russia.superjob.ru', 'Plastic and aluminum <b>windows</b>', 'New colleagues are already waiting for you! Easy search, free recommendations, fast import. · 20 years on the market. Recommendations. Easy search. Online support')
INSERT INTO ads VALUES('gorokna-11.ru', 'Plastic <b>windows</b> <b>in</b> Vorkuta. <b>Windows</b>, loggias, balconies', '<b>Windows</b> PVC with a discount in Vorkuta')
INSERT INTO ads VALUES('shop.oknagorizont.ru', 'Online store of <b>windows</b>, inexpensive plastic <b>windows</b>...', 'Online store of <b>windows</b> without installation, inexpensive plastic <b>windows</b> already today, low prices for <b>windows</b>. Many in stock and to order, urgent production...')
INSERT INTO ads VALUES('msk-oknaidveri.ru', 'Plastic <b>windows</b> PVC and entrance metal doors...', 'Classic PVC-<b>windows</b> - are <b>windows</b>, made of plastic, certified by the Ministry of Construction of Russia and complying with SNiP and others...')
INSERT INTO ads VALUES('oknamag.pro', 'Plastic <b>windows</b> for summer house buy at Oknamag Pro', 'Dimensions of plastic <b>window</b>: 600 height, 600 width. Available options... Quick view. Product code: opd0606.')
INSERT INTO ads VALUES('фурнитура-окон.рф', 'Hardware for plastic <b>windows</b> - Spare parts...', 'Hardware for <b>windows</b> <b>at</b> retail and wholesale. We will help you make the right choice of components and spare parts for your plastic <b>windows</b>.')
INSERT INTO ads VALUES('OknaGarant.ru', 'Ready-made plastic <b>windows</b> <b>in</b> <b>Moscow</b>, buy new...', '<b>Windows</b> are already waiting for you in our warehouse! The warehouse is located <b>in</b> <b>Moscow</b> within the MKAD. Savings, the cost is significantly lower (compared to <b>windows</b> to order).')
INSERT INTO ads VALUES('domaster.ru', 'Window accessories and fittings buy <b>in</b> <b>Moscow</b>...', 'For <b>windows</b> and doors. Brass online store — a place where you will buy accessories for caring for <b>windows</b> and fittings from different companies with delivery across Russia.')
INSERT INTO ads VALUES('sms-m.ru', 'Plastic <b>windows</b> manufacturing <b>in</b> <b>Moscow</b>', 'When the question of minimizing costs in the manufacture of modular buildings arose, StroyMontazhServis-M decided to launch its own...')
INSERT INTO ads VALUES('rosband.ru', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b>', 'Plastic <b>windows</b> possess excellent thermal insulation properties and there is no need to spend more ... <b>Moscow</b>, 1-1st Dobryninsky lane, 19 str. 6.')
INSERT INTO ads VALUES('moskva.miltor.ru', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b>: 185 offers', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b> you can by using the advertisements on our website. Here you will find out the cost of plastic <b>windows</b>, order terms and address...')
INSERT INTO ads VALUES('city-jaluzi.ru', 'Blinds for <b>windows</b> buy inexpensive <b>in</b> <b>Moscow</b> - reviews, photos...', 'We produce blinds for <b>windows</b>, sew curtains, assemble cornices and roller shutters to order since 2006. Prices from 495 to 5,900 ₽ per 1 sq.m, Free measurement and...')
Dump results to JSON
General output format:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.ads = [];
FOREACH item IN p1.ads;
obj.ads.push({
domain = item.domain
title = item.title
text = item.text
});
END;
obj.json %]
Initial text:
[
Final text:
]
Result example:
[{"ads": [{"domain":"oknamr.ru","text":"The company «<b>WINDOWS</b> <b>OF MOSCOW</b>» installs plastic <b>windows</b>, doors, and partitions, as well as balcony glazing <b>in</b> <b>Moscow</b> and the Moscow region.","title":"Plastic <b>windows</b> inexpensive <b>in</b> <b>Moscow</b>: buy at low..."},{"domain":"gor-okna.ru","text":"Order high-quality PVC <b>windows</b> <b>in</b> <b>Moscow</b> with delivery and installation. ... <b>Moscow</b>. from 9:00 to 21:00 daily.","title":"Buy PVC <b>windows</b> inexpensive | PVC <b>Windows</b> <b>in</b> <b>Moscow</b> at prices..."},{"domain":"balcon-msk.ru","text":"The company «Balconies <b>of Moscow</b>» produces professional glazing for balconies and loggias ... The price includes: Aluminum <b>windows</b> 3.2x0.7m h-1.5m, water ebb, installation...","title":"Glazing of balconies and loggias <b>in</b> <b>Moscow</b>."},{"domain":"oknamarket33.rf","text":"<b>Window</b> 1.3x1.4 for 9900 turnkey in a panel house! <b>Window</b> 2.0x1.4 for 13500 turnkey!","title":"Spring! Time to buy new PVC <b>windows</b>! Gift – Mosquito…"},{"domain":"okna-moskvy1.ru","text":"Only until the end of the month. REHAU <b>windows</b> with a discount of up to 60%. Multi-glass as a gift. Click!","title":"<b>Windows</b> <b>of Moscow</b>. Official site"},{"domain":"okno61.rf","text":"Plastic <b>windows</b> at low prices! Quality installation. Fast delivery. Warranty.","title":"<b>Windows</b> <b>in</b> Bataysk! Seasonal discounts"},{"domain":"gorokna-11.ru","text":"PVC <b>windows</b> with a discount in Vorkuta","title":"Plastic <b>windows</b> <b>in</b> Vorkuta. <b>Windows</b>, loggias, balconies"}]}]
To make the "Initial text" and "Final text" options available in the Task Editor, you need to activate "More options".
Possible settings
| Parameter | Default value | Description |
|---|---|---|
| Pages count | 10 | Number of pages to parse |
| Region of serp (lr=) | Moscow | Search region selection (lr= parameter) |
| AntiGate preset | default | Use of a captcha solving service. You must first configure the  Util::AntiGate parser - specify your access key and other parameters, then select the created preset here |