SE::DuckDuckGo - DuckDuckGo Search Results Scraper
Overview of the scraper
The DuckDuckGo search results scraper. allows you to obtain large databases of links ready for further use. You can use queries in the same way you enter them into the Dogpile search bar, including search operators (intitle, inurl, site etc.). More details on the official DuckDuckGo Search Syntax.
A-Parser's features allow you to save the parsing settings for the DuckDuckGo scraper for future use (presets), ), set a scraping schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations and lists to obtain the maximum possible number of results.
Results can be saved in the form and structure you need, thanks to the built-in powerful templater Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL and CSV.
Collected Data
- Links, anchors, and snippets from search results
Capabilities
- Supports all DuckDuckGo search operators(intitle:, inurl:, site: etc.). More details about search operators on the official DuckDuckGo Search Syntax
- Scrapes the maximum number of results provided by Bing - 10 pages with 10 items per result set
- Total number of results - 100
- Ability to scrape by selected location (Location option)
- Ability to select the results language (Language option)
Use Cases
- Collecting link bases - for A-Poster, XRumer, AllSubmitter, etc.
- Checking website indexing
- Finding backlinks (mentions) for websites
- Any other scenarios involving scraping DuckDuckGo in one way or another
Queries
Search phrases should be specified as queries, for example:
Football
test
site:a-parser.com
scraper site:a-parser.com
test -site:tests.com
IoT filetype:pdf
Query substitutions
You can use built-in macros to multiply queries, for example, we want to get a very large database of forums, we specify a few main queries in different languages:
forum
forum
foro
论坛
In the query format, we specify the iteration of characters from a to zzzz, , this method allows maximum rotation of search results and retrieval of many new unique results:
$query {az:a:zzzz}
This macro will create 475254 additional queries for each initial search query, which in total will give 4 x 475254 = 1901016 search queries, the number is impressive, but this is not a problem at all for A-Parser. With a speed of 2000 queries per minute, such a task will be processed in just 16 hours.
Using operators
You can use search operators in the query format, so it will be automatically added to each query in your list:
site:$query
Output results examples
A-Parser supports flexible results formatting thanks to the built-in templater Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured format, for example CSV or JSON
Exporting a list of links
Links + anchors + snippets with position output
Outputting links, anchors, and snippets to a CSV table
Saving related keywords
Checking links indexing
Saving in SQL format
Dumping results to JSON
Results Processing
A-Parser allows you to process results directly during scraping; in this section, we provide the most popular use cases for the DuckDuckGo scraper
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Link filtering by inclusion
Possible Settings
| Parameter Name | Default Value | Description |
|---|---|---|
| Pages count | 5 | Number of pages to scrape (from 1 to 10) |
| Region | US (English) | Location selection |
| Language | English (United States) | Language selection |
| Safe search | Moderate | Option to enable "Safe Search" |
| Serp time | Any time | Search Period |
| Use HTTP/2 | ☐ | Specifies whether to use HTTP/2 instead of HTTP/1.1 |
| User agent | Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0 | User-Agent header when requesting pages |