SE::Yandex::Direct - Yandex Direct ad scraper direct.yandex.ru
Scraper overview
SE::Yandex::Direct – Yandex Direct ads scraper. Automated collection of all data: number of ads for a specific query, title, text, domain, sitelinks and their anchors, tags.You can use automatic query multiplication, substitution of subqueries from files, enumeration of alphanumeric combinations and lists to get the maximum possible number of results. Using results filtering you can immediately clean the result by removing all unnecessary junk (using negative keywords).
The A-Parser functionality allows you to save scraping settings for the SE::Yandex::Direct scraper for future use (presets), set a scraping schedule, and much more.
Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
Collected data
- Number of ads for a specific query
- Title, text, and domain
- Sitelinks and their anchors
- Tags
- Whether the result is an advertisement
Capabilities
- Scrapes the maximum number of results provided by Yandex Direct – 10 pages
- Supports search region selection
- Ability to bypass captcha using the AntiCaptcha service or any other supporting their API (it is recommended to enable the is_russian parameter in the antigate settings)
Use Cases
- Collecting competitors advertising for given queries
Queries
Search phrases should be specified as queries, for example:
Buy a kite
Photoshop
Windows in Moscow
Apartment renovation
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to add a certain list of other words to each query, let's specify several main queries:
fantasy
tower defense
rpg
In the query format, we will specify a macro for substituting additional words from the keywords.txt file; this method allows increasing the query variability manifold:
{subs:keywords} $query
This macro will create as many additional queries as there are in the file for each original search query, which in total will give [number of original queries] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro.
For example, if the keywords.txt file contains:
free
online
As a result, the substitution macro will turn 3 main queries into 6:
free fantasy
online fantasy
free tower defense
online tower defense
free rpg
online rpg
Output results examples
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured one, such as CSV or JSON
Default output
Result format:
$query - $totalcount\n$ads.format('$domain: $title - $text\n')
Result example:
Photoshop - 5
www.syssoft.ru: Adobe Photoshop CS5 - Discounts - Photoshop CS5 with a discount from an authorized partner. Order now!
www.disted.ru: Learn to take photos! - Distance learning in Russian at the New York Institute of Photography.
www.softkey.ru: Licensed Adobe Photoshop CS! - Best new image editing features from Adobe Photoshop.
www.softprof-it.ru: Photoshop license from 23000 rubles - Low Photoshop price - from 23000 rubles. Call. Check purchase terms.
photoshop.pro1904.ru: Masterful work in Photoshop! - Unique course "Photoshop from scratch". Become a Photoshop Master.
Buy a kite - 0
Windows in Moscow - 33
www.imperia51.com: Windows in Murmansk! Low prices! - Very low prices! Warranty! Quality installation! All this is "Empire of Windows"!
mirperegorodok-kms.ru: "World of Partitions" Company - Aluminum systems: office partitions, entrance groups, doors...
inkar-avto.ru: Moscow-St. Petersburg-Moscow by bus - Mon, Tue, Wed, Thu - 1000 rub, Fri, Sat, Sun - 1300.
Output in CSV table
General result format:
[% FOREACH i IN p1.ads;
tools.CSVline(i.domain, i.title, i.text);
END %]
File name:
$datefile.format().csv
Initial text:
Domain,Title,Text
In the General result format, the Template Toolkit templater is used to output elements of the $ads array in a FOREACH loop.
In the result file name, you just need to change the file extension to csv.
To make the "Prepend text" option available in the Task Editor, you need to activate "More options". In "Prepend text", write the column names separated by commas and make the second line empty.
Saving in SQL format
Result format:
[% FOREACH ads;
"INSERT INTO ads VALUES('" _ domain _ "', '" _ title _ "', '" _ text _ "')\n";
END %]
Result example:
INSERT INTO ads VALUES('oknamr.ru', 'Plastic <b>windows</b> cheap <b>in</b> <b>Moscow</b>: buy at low...', 'The company «<b>WINDOWS</b> <b>OF MOSCOW</b>» performs installation of plastic <b>windows</b>, doors and partitions, as well as glazing of balconies <b>in</b> <b>Moscow</b> and Moscow region.')
INSERT INTO ads VALUES('gor-okna.ru', 'Buy <b>windows</b> PVC cheap | <b>Windows</b> PVC <b>in</b> <b>Moscow</b> at prices...', 'Order high-quality <b>windows</b> PVC <b>in</b> <b>Moscow</b> with delivery and installation. ... <b>Moscow</b>. from 9:00 to 21:00 daily.')
INSERT INTO ads VALUES('balcon-msk.ru', 'Glazing of balconies and loggias <b>in</b> <b>Moscow</b>.', 'Company "Balconies of <b>Moscow</b>» produces professional glazing of balconies and loggias ... Price includes: <b>Windows</b> aluminum 3.2x0.7m h-1.5m, water ebb, installation...')
INSERT INTO ads VALUES('windowsмаркет33.рф', 'Spring! Time to buy new <b>windows</b> PVC! Gift – Mosquito…', '<b>Window</b> 1.3x1.4 for 9900 turnkey in a panel house! <b>Window</b> 2.0x1.4 for 13500 turnkey!')
INSERT INTO ads VALUES('topokno72.ru', 'Plastic <b>windows</b>. Doors, balconies', 'Plastic <b>windows</b> and any structures made of PVC WinTech, Exprof, Proplex, Rehau, KBE')
INSERT INTO ads VALUES('okna-moskvy1.ru', '<b>Windows</b> <b>of Moscow</b>. Official site', 'Only until the end of the month. <b>Windows</b> REHAU with a discount up to 60%. Multi-glass as a gift. Click!')
INSERT INTO ads VALUES('russia.superjob.ru', 'Plastic and aluminum <b>windows</b>', 'New colleagues are already waiting for you! Convenient search, free recommendations, fast import. · 20 years on the market. Recommendations. Convenient search. Online support')
INSERT INTO ads VALUES('gorokna-11.ru', 'Plastic <b>windows</b> <b>in</b> Vorkuta. <b>Windows</b>,loggias,balconies', '<b>Windows</b> PVC with a discount in Vorkuta')
INSERT INTO ads VALUES('shop.oknagorizont.ru', 'Online store <b>of windows</b>, cheap plastic <b>windows</b>...', 'Online store <b>of windows</b> without installation, cheap plastic <b>windows</b> already today, low prices on <b>windows</b>. Many in stock and on order, urgent production...')
INSERT INTO ads VALUES('msk-oknaidveri.ru', 'Plastic <b>windows</b> PVC and entrance metal doors...', 'Classic PVC-<b>windows</b> - this is <b>windows</b>, made of plastic, certified by the Ministry of Construction of Russia and complying with SNiP and others...')
INSERT INTO ads VALUES('oknamag.pro', 'Plastic <b>windows</b> for summer house buy at Oknamag Pro', 'Dimensions of plastic <b>window</b>: 600 height, 600 width. Available options... Quick view. Product code: ord0606.')
INSERT INTO ads VALUES('фурнитура-windows.рф', 'Fittings for plastic <b>windows</b> - Spare parts...', 'Fittings for <b>windows</b> <b>in</b> retail and wholesale. We will help you make the right choice of components and spare parts for your plastic <b>windows</b>.')
INSERT INTO ads VALUES('OknaGarant.ru', 'Ready-made plastic <b>windows</b> <b>in</b> <b>Moscow</b>, buy new...', '<b>Windows</b> are already waiting for you in our warehouse! The warehouse is located <b>in</b> <b>Moscow</b> within the MKAD. Savings, the cost is significantly lower (compared to <b>windows</b> to order).')
INSERT INTO ads VALUES('domaster.ru', 'Window accessories and fittings buy <b>in</b> <b>Moscow</b>...', 'For <b>windows</b> and doors. Brass online store — a place where you will buy accessories for <b>windows</b> care and fittings from different companies with delivery across Russia.')
INSERT INTO ads VALUES('sms-m.ru', 'Plastic <b>windows</b> production <b>in</b> <b>Moscow</b>', 'When the question of minimizing costs in the production of modular buildings arose, StroyMontazhServis-M decided to launch its own...')
INSERT INTO ads VALUES('rosband.ru', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b>', 'Plastic <b>windows</b> have excellent thermal insulation properties and there is no need to spend more ... <b>Moscow</b>, 1-y Dobryninskiy pereulok, 19 str. 6.')
INSERT INTO ads VALUES('moskva.miltor.ru', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b>: 185 offers', 'Buy plastic <b>windows</b> <b>in</b> <b>Moscow</b> you can using the ads on our site. Here you will find out the cost of plastic <b>windows</b>, ordering terms and address...')
INSERT INTO ads VALUES('city-jaluzi.ru', 'Blinds for <b>windows</b> buy cheap <b>in</b> <b>Moscow</b> - reviews, photos...', 'We produce blinds for <b>windows</b>, sew curtains, assemble cornices and roller shutters to order since 2006 Prices from 495 to 5 900 ₽ per 1 sq.m, We measure for free and...')
Dump results to JSON
Common result format:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.ads = [];
FOREACH item IN p1.ads;
obj.ads.push({
domain = item.domain
title = item.title
text = item.text
});
END;
obj.json %]
Initial text:
[
Final text:
]
Result example:
[{"ads": [{"domain":"oknamr.ru","text":"Company "<b>WINDOWS</b> <b>OF MOSCOW</b>» performs installation of plastic <b>windows</b>, doors and partitions, as well as glazing of balconies <b>in</b> <b>Moscow</b> and Moscow region.","title":"Plastic <b>windows</b> cheap <b>in</b> <b>Moscow</b>: buy at low..."},{"domain":"gor-okna.ru","text":"Order high-quality <b>windows</b> PVC <b>in</b> <b>Moscow</b> with delivery and installation. ... <b>Moscow</b>. from 9:00 to 21:00 daily.","title":"Buy <b>windows</b> PVC cheap | <b>Windows</b> PVC <b>in</b> <b>Moscow</b> at prices..."},{"domain":"balcon-msk.ru","text":"Company "Balconies of <b>Moscow</b>» produces professional glazing of balconies and loggias ... Price includes: <b>Windows</b> aluminum 3.2x0.7m h-1.5m, water ebb, installation...","title":"Glazing of balconies and loggias <b>in</b> <b>Moscow</b>."},{"domain":"окнамаркет33.рф","text":"<b>Window</b> 1.3x1.4 for 9900 turnkey in a panel house! <b>Window</b> 2.0x1.4 for 13500 turnkey!","title":"Spring! Time to buy new <b>windows</b> PVC! Gift – Mosquito…"},{"domain":"okna-moskvy1.ru","text":"Only until the end of the month. <b>Windows</b> REHAU with a discount up to 60%. Multi-glass as a gift. Click!","title":"<b>Windows</b> <b>of Moscow</b>. Official site"},{"domain":"окно61.рф","text":"Plastic <b>windows</b> at low prices! Quality installation. Fast delivery. Warranty.","title":"<b>Windows</b> <b>in</b> Bataysk! Seasonal discounts"},{"domain":"gorokna-11.ru","text":"<b>Windows</b> PVC with a discount in Vorkuta","title":"Plastic <b>windows</b> <b>in</b> Vorkuta. <b>Windows</b>,loggias, balconies"}]}]
To make the "Prepend text" and "Append text" options available in the Task Editor, you need to activate "More options".
Possible settings
| Parameter | Default value | Description |
|---|---|---|
| Pages count | 10 | Number of pages to scrape |
| Region of serp (lr=) | Moscow | Search region selection (lr= parameter) |
| AntiGate preset | default | Use of a captcha solving service. It is necessary to pre-configure the  Util::AntiGate scraper - specify your access key and other parameters, then select the created preset here |