Skip to main content

SE::Dogpile - Dogpile search results scraper

img

Scraper Overview

The Dogpile search results scraper. With the SE::Dogpile scraper, you can obtain large databases of links ready for further use. You can use queries in the same format as you enter them in the Dogpile search bar, including search operators (url, language, site, etc.).

The A-Parser functionality allows you to save the parsing settings of the Dogpile scraper for further use (presets), set up a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to obtain the maximum possible number of results.

Results can be saved in the format and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Collected Data

  • Links, anchors, and snippets from the results
  • List of related keywords (Related keywords)
Collected data

Capabilities

  • Scrapes the maximum number of results returned by Dogpile - 50 pages of 10 items in the results
  • Total number of results - 500
  • Ability to search for related keywords

Use Cases

  • Collecting link databases - for A-Poster, XRumer, AllSubmitter, etc.
  • Competition assessment for keywords
  • Backlink (mention) search for websites
  • Website indexing check
  • Vulnerable website search
  • Any other use cases involving scraping Dogpile in one way or another

Queries

Queries should be specified as search phrases, just as if you were entering them directly into the Dogpile search form, for example:

test
parser language: ru
site: a-parser.com
site: slideshare.net Java Developer gmail.com resume -sample -samples -example -templates

Query Substitutions

You can use built-in macros to expand queries. For example, if we want to obtain a very large database of forums, we will specify several main queries in different languages:

forum
форум
foro
论坛

In the query format, we will specify the iteration of characters from a to zzzz. This method allows for maximum rotation of the search results and obtaining many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which will result in 4 x 475254 = 1901016 search queries in total, an impressive number, but not a problem for A-Parser. At a speed of 2000 requests per minute, this task will be completed in just 16 hours.

Using Operators

You can use search operators in the query format, so it will be automatically added to each query in your list:

site:$query

Output Results Examples

A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, allowing it to output results in any form, as well as in a structured format, such as CSV or JSON.

Exporting a list of links

Similar to SE::Google.

Similar to SE::Google.

Similar to SE::Google.

Similar to SE::Google.

Checking link indexing

Similar to SE::Google.

Saving in SQL format

Similar to SE::Google.

Dumping results to JSON

Similar to SE::Google.

Results processing

A-Parser allows processing results directly during scraping, in this section we have provided the most popular use cases for the Dogpile scraper

Similar to SE::Google.

Similar to SE::Google.

Extracting domains

Similar to SE::Google.

Removing tags from anchors and snippets

Similar to SE::Google.

Similar to SE::Google.

Possible settings

Parameter nameDefault valueDescription
Pages count10Number of pages to scrape (from 1 to 50)
Bypass CloudFlare with ChromeAutomatically bypass CloudFlare check
Bypass CloudFlare with Chrome Max Pages10Max. number of pages to bypass CF using Chrome
Bypass CloudFlare with Chrome HeadlessIf enabled, the browser will not be displayed during CF bypass using Chrome