SE::Rambler - Rambler search results scraper
Overview of the scraper
This Rambler search results scraper. allows you to obtain large databases of links ready for further use. You can use queries in the same format as you enter them into the Rambler search bar, including search operators (site, ip etc.).
A-Parser's functionality allows you to save Rambler scraper parsing settings for future use (presets), ), set up a parsing schedule, and much more. You can use automatic query multiplication, substitution of sub-queries from files, iteration over alphanumeric combinations and lists to get the maximum possible number of results.
Results can be saved in the format and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.
Collected Data
- Number of results in the search output
- Links, anchors, and snippets from the output
- List of related keywords (hints)
Capabilities
- Supports Rambler search operators (url:, site:, inurl:, host:, rhost:, domain:.)
- Scrapes up to 25 pages, with 10 to 50 results per page
- Scrapes related keywords ($hints)
- Option to use captcha-solving services
- Select the result device: regular desktop, mobile Android, or mobile iOS
Use Cases
- Collecting link databases
- Assessing competition for keywords
- Finding backlinks (mentions) for websites
- All cases where you need to scrape Rambler search results
Queries
Enter queries as you would in Rambler search. For example, if you need only links from one site. Enter the following in the query field:
"buy doors" site:http://kp.ru
Query substitutions
You can use built-in macros to multiply queries. For example, if we want to get a very large database of forums, we can specify several main queries in different languages:
forum
forum
foro
论坛
In the query format, we specify iterating through characters from a to zzzz, . This method maximizes search result rotation and yields many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each initial search query, which totals 4 x 475254 = 1901016 search queries. This impressive number is no problem for A-Parser. At a rate of 2000 queries per minute, this task will be processed in just 16 hours.
Using operators
You can use search operators in the query format, so it will be automatically added to every query in your list:
site:$query
Output Results Examples
A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in any desired form, as well as in structured formats like CSV or JSON
Exporting link list
Links + anchors + snippets with position output (loopcount)
Outputting links, anchors, and snippets to a CSV table
Saving related keywords
Result format:
$hints.format('$hint\n')
Result example:
habrahabr
habr
habrahabr ru
xabra
livebusiness
eureka
electronic accountant
elba island
elba electronic accountant
habrohabr
...
Saving in SQL format
Dump results to JSON
Results Processing
A-Parser allows results to be processed directly during scraping. In this section, we have listed the most popular use cases for the Rambler scraper
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Available Settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Device | Desktop | Select the result device: regular desktop, mobile Android, or mobile iOS |
| Pages count | 5 | Number of pages to scrape (from 1 to 25) |
| Links per page | 10 | Number of results per page (10/15/30/50) |
| Rambler region ID | Option to specify region. Requires region ID. How to find the required ID is described here | |
| Sort | Sites by relevance | Select the result sorting option |
| Results filtering | Moderate | Select the result filtering option |
| Results language | Any language | Select the search language for results |
| Serp time | Anytime | Select the result period |
| Results type | Any format | Select the result type (mime type) |
| Exact match | ☐ | Strict query matching |
| Disable autocorrect | ☐ | Disables autocorrection, allowing scraping the output exactly for the specified query |
| Use sessions | ☑ | Saves good sessions, allowing faster scraping with fewer errors |
| AntiGate preset | default | Determines whether to use  Util::AntiGate to bypass captchas |