Skip to main content

SE::Dogpile - Dogpile search results parser

img

Dogpile parser overviewโ€‹

Dogpile search results parser. With the SE::Dogpile parser, you can get large databases of links ready for further use. You can use queries in the same format as you enter them in the Dogpile search box, including search operators (url, language, site, etc.).

The A-Parser functionality allows you to save the parsing settings of the Dogpile parser for further use (presets), set up a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, enumeration of alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the format and structure that you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

List of collected dataโ€‹

  • Links, anchors, and snippets from search results
  • List of related keywords (Related keywords)

Data collected by the SE::Dogpile parser

Capabilitiesโ€‹

  • Parses the maximum number of results returned by Dogpile - 50 pages of 10 items in the results
  • Total number of results - 500
  • Ability to search for related keywords

Usage scenariosโ€‹

  • Collecting link databases - for A-Poster, XRumer, AllSubmitter, etc.
  • Keyword competition assessment
  • Finding backlinks (mentions) of sites
  • Checking site indexing
  • Finding vulnerable sites
  • Any other scenarios involving parsing Dogpile in one form or another

Query examplesโ€‹

  • Queries must be specified as search phrases, just as if you were entering them directly into the Dogpile search form, for example:
test
parser language: ru
site: a-parser.com
site: slideshare.net Java Developer gmail.com resume -sample -samples -example -templates

Query substitutionsโ€‹

You can use built-in macros to multiply queries. For example, we want to get a very large database of forums, so we specify several basic queries in different languages:

forum
ั„ะพั€ัƒะผ
foro
่ฎบๅ›

In the query format, we specify the enumeration of characters from a to zzzz. This method allows you to rotate the search results to the maximum and get many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which will give a total of 4 x 475254 = 1901016 search queries. This is an impressive number, but it is not a problem for A-Parser. At a speed of 2000 requests per minute, this task will be processed in just 16 hours.

Using operatorsโ€‹

You can use search operators in the query format, so it will be automatically added to each query from your list:

site:$query

Result output optionsโ€‹

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in any form, as well as in a structured form, such as CSV or JSON.

Same as in SE::Google.

Same as in SE::Google.

Same as in SE::Google.

Same as in SE::Google.

Same as in SE::Google.

Saving in SQL formatโ€‹

Same as in SE::Google.

Dumping results in JSONโ€‹

Same as in SE::Google.

Result processingโ€‹

A-Parser allows you to process results during parsing, and in this section, we have provided the most popular use cases for the Dogpile parser.

Same as in SE::Google.

Same as in SE::Google.

Extracting domainsโ€‹

Same as in SE::Google.

Removing tags from anchors and snippetsโ€‹

Similarly to SE::Google.

Similarly to SE::Google.

Possible settingsโ€‹

Parameter nameDefault valueDescription
Pages count10Number of pages to parse (from 1 to 50)
Util::ReCaptcha2 presetdefaultDetermines whether to use Util::ReCaptcha2Util::ReCaptcha2 to bypass captchas
ReCaptcha2 retries3Number of attempts to send a response for reCAPTCHA, specified number of times without changing the proxy