SE::Dogpile - Dogpile Search Results Scraper
Overview of the scraper
Dogpile search results scraper. Thanks to the SE::Dogpile scraper, you can get large link databases ready for further use. You can use queries in the same form you enter them into the Dogpile search bar, including search operators (url, language, site etc.).
A-Parser functionality allows you to save Dogpile scraper settings for future use (presets), set a scraping schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to get the maximum possible number of results.
Results can be saved in the form and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.
Collected data
- Links, anchors and snippets from the search results
- List of related keywords
Capabilities
- The scraper parses the maximum number of results provided by Dogpile - 50 pages with 10 items in the search results
- Total number of results - 500
- Ability to search for related keywords
Use cases
- Collecting link bases - for A-Poster, XRumer, AllSubmitter, etc.
- Assessing competition for keywords
- Finding backlinks (mentions) of websites
- Checking site indexing
- Finding vulnerable websites
- Any other options involving Dogpile scraping in one way or another
Queries
As queries, you must specify search phrases, just as if you were entering them directly into the Dogpile search form, for example:
test
parser language: ru
site: a-parser.com
site: slideshare.net Java Developer gmail.com resume -sample -samples -example -templates
Query substitutions
You can use built-in macros to multiply queries, for example, we want to get a very large database of forums, we will specify several main queries in different languages:
forum
forum
foro
论坛
In the query format, we will specify the iteration of characters from a to zzzz, this method allows maximum rotation of search results and retrieval of many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each initial search query, which will total 4 x 475254 = 1901016 search queries, the number is impressive, but it's not a problem for A-Parser. At a speed of 2000 queries per minute, this task will be processed in just 16 hours.
Using operators
You can use search operators in the query format, so it will be automatically added to each query in your list:
site:$query
Output results examples
A-Parser supports flexible results formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in arbitrary form, as well as in a structured form, such as CSV or JSON
Exporting a list of links
Links + anchors + snippets with position output
Output of links, anchors and snippets into a CSV table
Saving related keywords
Checking links indexing
Saving in SQL format
Dumping results to JSON
Results processing
A-Parser allows processing results directly during scraping, in this section we have provided the most popular use cases for the Dogpile scraper
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Possible settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Pages count | 10 | Number of pages to scrape (from 1 to 50) |
| Bypass CloudFlare with Chrome | ☑ | Automatic bypass of CloudFlare check |
| Bypass CloudFlare with Chrome Max Pages | 10 | Max. number of pages when bypassing CF via Chrome |
| Bypass CloudFlare with Chrome Headless | ☑ | If enabled, the browser will not be displayed during CF bypass via Chrome |