SE::Bing - Bing Search Results Scraper
Scraper Overview
The Bing search results scraper. Thanks to the SE::Bing scraper, you can obtain huge databases of links ready for further use. You can use queries in the same way as you enter them in the Bing search bar, including search operators (contains, intitle, language, site, etc.). Learn more on the official page Advanced search keywords.
A-Parser functionality allows you to save Bing scraper parsing settings for future use (presets), set a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to obtain the maximum possible number of results.
Saving results is possible in the form and structure that you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
Use Cases for the Scraper
🔗 Scraping Bing search results
Scraping Bing with automatic query multiplication (Parse all results)
🔗 Scraping top 10 and content of tags
Multi-level scraping. Parsing the top 10 and the content of title and description tags for these sites
🔗 JS scraper based on Bing scraper
Creating JS scrapers. Getting results from the standard scraper
🔗 Scraping all results in Bing
Scraping with intelligent query substitution through the Template Toolkit and tools.query.add
🔗 Unique URLs in Bing
Scraping unique URLs by keyword and uniqueness by domain, with the ability to specify the maximum number of URLs on one domain
🔗 Multiple scrapers in a preset
The preset shows an example of using multiple scrapers
Collected Data
- Number of results per query
- Links, anchors, and snippets from the search results
- List of related keywords (Related keywords)
- Links, visible links, anchors, snippets, and ad position from the ad results
Capabilities
- Support for all Bing search operators (site:, ip:, etc.). Learn more about search operators on the official page Advanced search keywords.
- Scrapes the maximum number of results Bing can return - 50 pages of 50 items each in the search results
- Can automatically scrape more than 1000 results per query - adds additional characters (option Parse all results)
- Ability to deeply scrape by related keywords (Parse related to level)
- Ability to search for related keywords
- Ability to scrape links to cached pages
- Ability to scrape mobile search results
The following scrapers are based on the Bing scraper:
SE::Bing::Position - determining the positions of any websites in the search results for a list of queries
Use Cases
- Collecting a database of links - for A-Poster, XRumer, AllSubmitter, etc.
- Competition assessment for keywords
- Searching for backlinks (mentions) of websites
- Checking website indexing
- Searching for vulnerable websites
- Searching for websites on the same IP address
- Any other scenarios involving scraping Bing in one way or another
Queries
As queries, you need to specify search phrases, just as if they were entered directly into the Bing search form, for example:
test
окна Москва
site:http://lenta.ru
ip:222.36.12.12
Query Substitutions
You can use built-in macros to diversify queries, for example, if we want to get a very large database of forums, we will specify several basic queries in different languages:
forum
форум
foro
论坛
In the query format, we will specify a character permutation from a to zzzz, this method allows to maximally rotate the search output and get many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries, an impressive figure, but this is not a problem for A-Parser. With a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Using Operators
You can use search operators in the query format, so it will be automatically added to each query from your list:
site:$query
Output Results Examples
A-Parser supports flexible formatting of results thanks to the built-in Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON
Exporting a List of Links
Links + Anchors + Snippets with Position Output
Outputting Links, Anchors, and Snippets in a CSV Table
Saving Related Keywords
Keyword Competition
Checking Link Indexing
Saving in SQL Format
Dumping Results to JSON
Results Processing
A-Parser allows you to process results directly during scraping, in this section we have provided the most popular cases for the Bing scraper
Link Deduplication
Link Deduplication by Domain
Extracting Domains
Removing Tags from Anchors and Snippets
Filtering Links by Inclusion
Possible Settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Pages count | 10 | Number of pages for scraping (from 1 to 50) |
| Links per page | 30 | Number of links in the output for each page (Auto / 10 / 15 / 30 / 50) |
| Region | Based on IP | Region selection. List of regions. |
| Interface language | Any | Interface language selection. List of languages. |
| Safe Search | Moderate | Safe search option selection (Strict / Moderate / Off) |
| Device | Desktop | Device selection for search (Desktop / Mobile) |
| Use sessions | ☑ | Saves good sessions, allowing for faster scraping with fewer errors. |
| Fix pagination bug | ☑ | Fixes a bug in Bing search that results in empty 2nd and subsequent pages |