SE::DuckDuckGo - DuckDuckGo search engine scraper
Overview of the parser
The DuckDuckGo search results parser. Thanks to the DuckDuckGo parser, you can obtain large databases of links ready for further use. You can use queries in the same form as you enter them into the Dogpile search bar, including search operators (intitle, inurl, site, etc.). More details on the official DuckDuckGo Search Syntax page.
A-Parser functionality allows you to save DuckDuckGo parser settings for future use (presets), set parsing schedules, and much more. You can use automatic query multiplication, substitution of subqueries from files, brute-forcing of alphanumeric combinations and lists to obtain the maximum possible number of results.
Saving results is possible in the form and structure you need, thanks to the built-in powerful Template Toolkit template engine, which allows applying additional logic to results and outputting data in various formats, including JSON, SQL, and CSV.
Collected data
- Links, anchors, and snippets from the search results
Capabilities
- Support for all DuckDuckGo search operators (intitle:, inurl:, site:, etc.). More details about search operators on the official DuckDuckGo Search Syntax page
- Parses the maximum number of results provided by Bing - 10 pages of 10 elements in the search results
- Total number of results - 100
- Ability to parse by selected location (Location option)
- Ability to select the search results language (Language option)
Use cases
- Collecting link databases - for A-Poster, XRumer, AllSubmitter, etc.
- Checking website indexing
- Searching for backlinks (mentions) of websites
- Any other cases involving DuckDuckGo parsing in one form or another
Queries
You should specify search phrases as queries, for example:
Football
test
site:a-parser.com
parser site:a-parser.com
test -site:tests.com
IoT filetype:pdf
Query substitutions
You can use built-in macros for query multiplication; for example, if we want to get a very large database of forums, we specify several main queries in different languages:
forum
forum
foro
论坛
In the query format, we specify a character brute-force from a to zzzz; this method allows for maximum rotation of search results and obtaining many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each original search query, which in total will give 4 x 475254 = 1901016 search queries—an impressive figure, but no problem at all for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Using operators
You can use search operators in the query format, so they will be automatically added to each query from your list:
site:$query
Output results examples
A-Parser supports flexible result formatting thanks to the built-in Template Toolkit template engine, which allows it to output results in arbitrary forms, as well as structured ones like CSV or JSON
Exporting a list of links
Links + anchors + snippets with position output
Outputting links, anchors, and snippets to a CSV table
Saving related keywords
Checking link indexing
Saving in SQL format
Dumping results to JSON
Results processing
A-Parser allows processing results directly during parsing; in this section, we have listed the most popular cases for the DuckDuckGo parser
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Possible settings
| Parameter name | Default value | Description |
|---|---|---|
| Pages count | 5 | Number of pages to parse (from 1 to 10) |
| Region | US (English) | Location selection |
| Language | English (United States) | Language selection |
| Safe search | Moderate | Ability to enable "Safe Search" |
| Serp time | Any time | Search period |
| Use HTTP/2 | ☐ | Determines whether to use HTTP/2 instead of HTTP/1.1 |
| User agent | Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0 | User-Agent header when requesting pages |