SE::Yahoo - Yahoo Search Results Scraper
Overview
The Yahoo search results scraper. Thanks to the Yahoo scraper, you can obtain large databases of links ready for further use. You can use queries in the same way you enter them in the Yahoo search bar, including search operators (site, ip, etc.).
A-Parser functionality allows you to save Yahoo scraper parsing settings for future use (presets), set a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results.
Saving results is possible in the form and structure that you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
Collected Data
- Links, anchors, and snippets from the results
- List of related keywords (Related keywords)
- Ad results
Capabilities
- Support for all Yahoo search operators (site:, ip:, etc.)
- Scraps the maximum number of results Yahoo returns - 50 pages of 100 items each in the results
- Can automatically scrape more than 1000 results per query - by adding additional characters (option Parse all results)
- Ability to deeply scrape by related keywords (Parse related to level)
- Ability to search for related keywords
- Supports specifying the time of the results
Use Cases
- Collecting link databases - for A-Poster, XRumer, AllSubmitter, etc.
- Competition assessment for keywords
- Searching for backlinks (mentions) of websites
- Checking website indexing
- Searching for websites on the same IP address
- Searching for vulnerable websites
- Any other scenarios involving scraping Yahoo in one way or another
Queries
As queries, you need to specify search phrases, just as if you were entering them directly into the Yahoo search form, for example:
test
окна Москва
site:http://lenta.ru
ip:222.36.12.12
Query Substitutions
You can use built-in macros for query multiplication, for example, if we want to get a very large database of forums, we will specify several main queries in different languages:
forum
форум
foro
论坛
In the query format, we will specify a character iteration from a to zzzz, this method allows for maximum rotation of the search results and obtaining many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries, an impressive figure, but this is not a problem for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Using Operators
You can use search operators in the query format, so it will be automatically added to each query from your list:
site:$query
Output Results Examples
A-Parser supports flexible formatting of results thanks to the built-in Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Export list of links
Links + anchors + snippets with position output
Output of links, anchors, and snippets in a CSV table
Saving related keywords
Keyword competition
Checking link indexing
Saving in SQL format
Dump results to JSON
Results processing
A-Parser allows you to process results directly during scraping, in this section we have listed the most popular cases for the Yahoo scraper
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Possible settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Pages count | 5 | Number of pages to scrape (from 1 to 50) |
| Serp time | All time | Serp time (time-dependent search, "tbs=" parameter: All time / Past 24 hours / Past week / Past month) |
| Safe Search | Moderate | Choice of Safe Search option (Off / Moderate / Strict) |
| Yahoo domain | United States (English) | Choice of Yahoo domain |
| Yahoo language | Any | Choice of Yahoo language, allows you to select the language for the search |
| Yahoo country | Any | Choice of country, allows you to select the country for which the search will be conducted |
| Not found is error | ☐ | Whether to consider the absence of results as an error |