Skip to main content

SE::Rambler - Rambler search engine scraper

img

Scraper Overview​

Rambler search engine results scraper. Thanks to the Rambler scraper, you can obtain large databases of links ready for further use. You can use queries in the same format as you enter them in the Rambler search bar, including search operators (site, etc.).

A-Parser functionality allows you to save Rambler scraper settings for future use (presets), set scraping schedules, and much more. You can use automatic query multiplication, substitution of subqueries from files, and iteration of alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to results and output data in various formats, including JSON, SQL, and CSV.

Collected data​

  • $totalcount — number of results in the SERP. From the first page only; empty if the number is missing
  • $serp.$i.link, $serp.$i.anchor, $serp.$i.snippet — link, anchor, snippet
  • $hints.$i.hint — related keywords. From the first page only; empty if there are no hints

what data the SE::Rambler scraper collects

Capabilities​

  • Support for Rambler search operators (url:, site:, inurl:, host:, rhost:, domain:)
  • Up to 25 pages, no more than ~250 results
  • Related keywords ($hints.$i.hint)
  • SERP device types: desktop, mobile Android, or mobile iOS

Use Cases​

  • Collecting link databases
  • Assessing competition for keywords
  • Searching for site backlinks (mentions)
  • All cases where you need to scrape Rambler search results

Queries​

Specify queries just as in Rambler search. For example, if you only need links from one site. Enter in the query field:

"buy doors" site:http://kp.ru

Query substitutions​

You can use built-in macros for query multiplication; for example, if we want to get a very large database of forums, we specify several main queries in different languages:

forum
forum
foro
论坛

In the query format, we specify a character iteration from a to zzzz; this method allows for maximum rotation of search results and obtaining many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries—an impressive figure, but no problem at all for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.

Using operators​

You can use search operators in the query format, so they will be automatically added to each query from your list:

site:$query

Output results examples​

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in arbitrary form, as well as structured formats like CSV or JSON.

Exporting a list of links​

Same as in SE::Google.

Same as in SE::Google.

Same as in SE::Google.

Result format:

$hints.format('$hint\n')

Example result:

habrahabr
habr
habrahabr ru
xabra
livebusiness
eureka
electronic accountant
elba island
elba electronic accountant
habrahabr
...

Saving in SQL format​

Same as in SE::Google.

Dumping results to JSON​

Same as in SE::Google.

Results processing​

A-Parser allows processing results directly during scraping; in this section, we have listed the most popular cases for the Rambler scraper.

Same as in SE::Google.

Same as in SE::Google.

Extracting domains​

Same as in SE::Google.

Removing tags from anchors and snippets​

Same as in SE::Google.

Same as in SE::Google.

Possible settings​

note

General settings for all scrapers

Parameter nameDefault valueDescription
DeviceDesktopOutput device: desktop, mobile Android, or mobile iOS
Wait for soft captcha, sec (0 = ban)25How many seconds to wait for soft captcha completion. 0 — ban proxy immediately
Page load timeout, sec20Page load timeout
Pages count5How many pages to scrape, 1–25. Rambler provides no more than ~250 results, so the maximum number of pages depends on Links per page: 10 results → 25 pages, 15 → 17 pages, 30 → 8 pages, 50 → 5 pages.
Links per page10Results per page: 10 / 15 / 30 / 50
Rambler region IDRegion ID. Non-CIS IP — Moscow. CIS IP — results by IP or by this ID if specified. How to find the ID — here
SortSites by relevanceSorting: Pages by relevance, Pages by date, Sites by relevance, Sites by date
Results filteringModerateFilter: Unrestricted, Moderate, Family search
Results languageAny languageResults language: any or Belarusian, English, German, French, Kazakh, Russian, Tatar, Turkish, Ukrainian
Serp timeAnytimePeriod: Anytime, Past 24 hours, Past week, Past month
Results typeAny formatDocument type: any or pdf, doc, rtf, xls, ppt, swf, odt, ods, odp, odg
Exact match☐Exact match to the query
Disable autocorrect☐Disables autocorrection, scrapes results for the specified query
Max concurrent browser pages10How many browser tabs the scraper keeps open simultaneously. Increase carefully with high thread counts — memory and CPU load will grow
Use sessions☑Saves good sessions, fewer errors, faster scraping
Headless browser☑Browser without a window. Uncheck for debugging