SE::Dogpile - Dogpile search results parser
Dogpile parser overviewโ
Dogpile search results parser. With the SE::Dogpile parser, you can get large databases of links ready for further use. You can use queries in the same format as you enter them in the Dogpile search box, including search operators (url, language, site, etc.).
The A-Parser functionality allows you to save the parsing settings of the Dogpile parser for further use (presets), set up a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, enumeration of alphanumeric combinations and lists to obtain the maximum possible number of results.
Saving results is possible in the format and structure that you need, thanks to the built-in powerful Template Toolkit template engine, which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.
List of collected dataโ
- Links, anchors, and snippets from search results
- List of related keywords (Related keywords)
Capabilitiesโ
- Parses the maximum number of results returned by Dogpile - 50 pages of 10 items in the results
- Total number of results - 500
- Ability to search for related keywords
Usage scenariosโ
- Collecting link databases - for A-Poster, XRumer, AllSubmitter, etc.
- Keyword competition assessment
- Finding backlinks (mentions) of sites
- Checking site indexing
- Finding vulnerable sites
- Any other scenarios involving parsing Dogpile in one form or another
Query examplesโ
- Queries must be specified as search phrases, just as if you were entering them directly into the Dogpile search form, for example:
test
parser language: ru
site: a-parser.com
site: slideshare.net Java Developer gmail.com resume -sample -samples -example -templates
Query substitutionsโ
You can use built-in macros to multiply queries. For example, we want to get a very large database of forums, so we specify several basic queries in different languages:
forum
ัะพััะผ
foro
่ฎบๅ
In the query format, we specify the enumeration of characters from a to zzzz. This method allows you to rotate the search results to the maximum and get many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each original search query, which will give a total of 4 x 475254 = 1901016 search queries. This is an impressive number, but it is not a problem for A-Parser. At a speed of 2000 requests per minute, this task will be processed in just 16 hours.
Using operatorsโ
You can use search operators in the query format, so it will be automatically added to each query from your list:
site:$query
Result output optionsโ
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in any form, as well as in a structured form, such as CSV or JSON.
Exporting a list of linksโ
Links + anchors + snippets with position outputโ
Output of links, anchors, and snippets in a CSV tableโ
Saving related keywordsโ
Checking link indexingโ
Saving in SQL formatโ
Dumping results in JSONโ
Result processingโ
A-Parser allows you to process results during parsing, and in this section, we have provided the most popular use cases for the Dogpile parser.
Link uniquenessโ
Link uniqueness by domainโ
Extracting domainsโ
Removing tags from anchors and snippetsโ
Filtering links by inclusionโ
Possible settingsโ
| Parameter name | Default value | Description |
|---|---|---|
| Pages count | 10 | Number of pages to parse (from 1 to 50) |
| Util::ReCaptcha2 preset | default | Determines whether to use  Util::ReCaptcha2 to bypass captchas |
| ReCaptcha2 retries | 3 | Number of attempts to send a response for reCAPTCHA, specified number of times without changing the proxy |