Skip to main content

SE::Dogpile - Dogpile search engine scraper

img

Overview of the parser​

Dogpile search results parser. Thanks to the SE::Dogpile parser, you will be able to obtain large databases of links ready for further use. You can use queries in the same form as you enter them in the Dogpile search bar, including search operators (url, language, site, etc.).

The A-Parser functionality allows you to save Dogpile parser settings for future use (presets), set a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, brute-forcing of alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Collected data​

  • Links, anchors, and snippets from the search results
  • List of related keywords
Collected data

Capabilities​

  • Parses the maximum number of results provided by Dogpile - 50 pages of 10 items in the search results
  • Total number of results - 500
  • Ability to search for related keywords

Use cases​

  • Collecting link databases - for A-Poster, XRumer, AllSubmitter, etc.
  • Assessing competition for keywords
  • Searching for backlinks (mentions) of sites
  • Checking site indexing
  • Searching for vulnerable sites
  • Any other options involving Dogpile parsing in one form or another

Queries​

As queries, you must specify search phrases exactly as if they were entered directly into the Dogpile search form, for example:

test
parser language: ru
site: a-parser.com
site: slideshare.net Java Developer gmail.com resume -sample -samples -example -templates

Query substitutions​

You can use built-in macros for query multiplication; for example, we want to get a very large database of forums, let's specify several main queries in different languages:

forum
forum
foro
论坛

In the query format, we will specify a character brute-force from a to zzzz; this method allows for maximum rotation of search results and obtaining many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries; the figure is impressive, but it is not a problem at all for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.

Using operators​

You can use search operators in the query format, so it will be automatically added to each query from your list:

site:$query

Output results examples​

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in any form, as well as in structured formats like CSV or JSON

Exporting a list of links​

Same as in SE::Google.

Same as in SE::Google.

Same as in SE::Google.

Same as in SE::Google.

Checking link indexing​

Same as in SE::Google.

Saving in SQL format​

Same as in SE::Google.

Dumping results to JSON​

Same as in SE::Google.

Results processing​

A-Parser allows processing results directly during parsing; in this section, we have listed the most popular cases for the Dogpile parser

Same as in SE::Google.

Same as in SE::Google.

Extracting domains​

Same as in SE::Google.

Removing tags from anchors and snippets​

Same as in SE::Google.

Same as in SE::Google.

Possible settings​

note

Common settings for all parsers

Parameter nameDefault valueDescription
Pages count10Number of pages to parse (from 1 to 50)
Bypass CloudFlare with Chrome☑Automatic CloudFlare check bypass
Bypass CloudFlare with Chrome Max Pages10Max. number of pages when bypassing CF via Chrome
Bypass CloudFlare with Chrome Headless☑If the option is enabled, the browser will not be displayed during CF bypass via Chrome