SE::DuckDuckGo::Position - Checking website positions by keywords in DuckDuckGo
Overview of the scraper
Site position checking scraper by keywords in DuckDuckGo. Thanks to the SE::DuckDuckGo::Position scraper, you can automatically check positions in DuckDuckGo search results using your own domain databases. Using the SE::DuckDuckGo::Position scraper, you can easily, accurately, and quickly determine the position of a site in DuckDuckGo.
A-Parser's functionality allows you to save the scraping settings of the SE::DuckDuckGo::Position scraper for future use (presets), set a scraping schedule, and much more. You can use automatic substitution of subqueries from files.
Saving results is possible in the form and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL and CSV.
Collected data
- Site position and link to the site page
- List of all site positions and links to pages
Capabilities
- All capabilities of the scraper
SE::DuckDuckGo - Automatically stops scraping when the site is found
- Supports subdomain search
- Ability to compare the sought position by domain, top-level domain, and full link
- Collecting positions for several domains simultaneously
Use cases
- Checking the positions of your sites and competitor sites
- Finding traffic pages of a site
Queries
As queries, you must specify the domain of the desired site and the search query separated by a space, for example:
lenta.ru новости
lenta.ru новости онлайн
ria.ru новости
a-parser.com А-Парсер
If you need to check one site against a list of queries, you can specify the domain in the query format (Query format):
lenta.ru $query
Or just use a list of keywords. To use several domains in the query at once, you need to specify a list of domains separated by a comma and the keyword separated by a space, for example:
lenta.ru,ria.ru,notfound.com новости лента
The results will be recorded in the $bulkcheck array.
The Stop when found option is also supported, scraping will end if positions are found for all domains.
Query substitutions
You can use built-in macros for automatic substitution of subqueries from files, for example, we want to check sites/a site against a keyword database, let's specify a few main queries:
ria.ru
lenta.ru
rbc.ru
yandex.ru
In the query format, we will specify the macro for substituting additional words from the file Keywords.txt, this method allows you to check a domain database against a keyword database and obtain positions as a result:
$query {subs:Keywords}
This macro will create as many additional queries as there are in the file for each initial search query, which in total will be [number of initial queries (domains)] x [number of queries in the Keywords file] = [total number of queries] as a result of the macro operation.
Output results examples
A-Parser supports flexible formatting of results thanks to the built-in templating engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured format, such as CSV or JSON
Exporting a list of positions
Getting the result in the form:
searched domain - key: position number in the SERP
Result format:
$domain - $key: $position\n
Example result:
lenta.ru - новости онлайн: 13
lenta.ru - новости: 26
ria.ru - новости: 1
a-parser.com - А-Парсер: 1
...
Simultaneous check of several domains (bulk check)
Information on all domains when checking several domains simultaneously is contained in the $bulkcheck array.
Result format:
$bulkcheck.format('$domain - $position\n')
Query example:
lenta.ru,ria.ru,notfound.com новости лента
Example result:
lenta.ru - 2
ria.ru - 6
notfound.com - 0
Links + anchors + snippets with position output
Outputting links, anchors and snippets to a CSV table
Saving related keywords
Keyword competition
Checking link indexing
Saving in SQL format
Dumping results to JSON
Results processing
A-Parser allows you to process results directly during scraping, in this section we have provided the most popular use cases for the SE::DuckDuckGo::Position scraper
Saving domains without zero positions
The example of simultaneous checking of several domains was taken as a basis (see above in the output options) and a filter was added.
Add a filter and select the position output variable from the drop-down list. Select type: >. Then you need to write 0 in Number. With this filter, you can remove all results with a zero position.
Download example
How to import an example into A-Parser
eJx1VNtu2zAM/RVDCNAVyIJ2a4HBDwPSbhk2dE3Wy1OSB8WiWy2y6OmSXoL8+yhZ
ttNufYgi3g55SMpb5rhd25kBC86yfL5ldbyznF1/zfMvvliH3zfM8xla6STq7H12
zTeQCay41DZ7kO4evct49gwGszq5sSGrubFgAuz8TTRyE1Byrxwbbpl7qoFS4waM
kQLIKAXJ3kJt8PGJ5A1XnlxKrizs3o4o0VTcEZUI3EaxwcqrdXEPxXrUeLw7GDQ8
iNWgLX2x0AeHbA+9lMqBIZwEmM9ZBxR4tlyWw+R608R93kt9RHeso1vOLGjLdstl
i2gnsZpQYX08SiPojKHdN9jUAb16QtIlryIvwR0Ea0vrcOQeAwIXIpbGVZMhDKPP
eqvln1icRvKlq5FgJwYrUjmIAEH51FY3Z4MoB6I+xv5qYtJAhsxSqRNOhYjXFklt
4Q7NNPaA9FuGeqzUBWxA9W4R/8xLJWhzxiUFfU+B/3eZ/oOx6+jtp6L9eDBUQ4cS
pbPpzz5K4AXeEXOxIt5KVtKRbM/Ra5fmtwaou55dhp5VaKBLk5BTdnpUNeiwjP3I
xnWvekHjxVheKgvUpbybpgVvPb2+oZc71edY1QoCL+2VGoanctWvx9imMQShL/B1
8HlMEai3T5E5RGV/XDel1kbS+p2GAivq5H7WBFlwpW6vLvYtrF8pEhRox0fGD42M
fxpdSZ0VowKrbOGPTj6KeEI8P/T3k+N4NppPjesqnqd9WDKHGRX0Eu6QdpT6tFt2
H6HuE7d9+1OUb3e0Br/trAkJPQsBpKPm2+hxvPsLSinNVw==
See also: Result filters
Link deduplication
Link deduplication by domain
Extracting domains
Removing tags from anchors and snippets
Filtering links by inclusion
Possible settings
Supports all scraper settings SE::DuckDuckGo, as well as additionally:
| Parameter name | Default value | Description |
|---|---|---|
| Result format | $domain - $key: $position\n | Default result output format |
| Stop when found | ☑ | Stop scraping if a domain is found, it will not go to the next pages |
| Match type | Exact domain | Ability to compare the desired position by domain, by top-level domain, and by full link (Exact domain / Top level domain / Exact url) |