Net::DNS - Scraper Resolves Domains to IP Addresses
Overview of the scraper
Net::DNS – collects domain name DNS records. Automated bulk resolving of domain databases, checking domain availability.A-Parser functionality allows saving Net::DNS scraper parsing settings for future use (presets), ), scheduling parsing, and much more.
Results can be saved in the format and structure you need, thanks to the built-in powerful template engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.
Collected Data
- List of IP addresses
- DNS server used on successful request
- DNS record of any type (outputted as JSON, making it convenient to retrieve their individual parameters)
Use Cases
- Resolving a domain database
- Checking domain availability
Queries
The queries should be a list of domains, for example:
a-parser.com
yandex.ru
google.com
vk.com
facebook.com
youtube.com
Output Results Examples
A-Parser supports flexible result formatting thanks to the built-in template engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in structured formats, such as CSV or JSON
Default output
Result format:
$query: $ip ($ips.format('$ip, '))\n
The result shows the domain and its IP addresses:
youtube.com: 173.194.70.136 (173.194.70.136, 173.194.70.190, 173.194.70.93, 173.194.70.91, )
google.com: 46.28.247.98 (46.28.247.98, 46.28.247.89, 46.28.247.84, 46.28.247.108, 46.28.247.119, 46.28.247.114, 46.28.247.88, 46.28.247.94, 46.28.247.99, 46.28.247.109, 46.28.247.93, 46.28.247.104, 46.28.247.113, 46.28.247.118, 46.28.247.123, 46.28.247.103, )
facebook.com: 173.252.110.27 (173.252.110.27, )
a-parser.com: 176.9.9.90 (176.9.9.90, )
vk.com: 87.240.143.244 (87.240.143.244, 87.240.131.120, )
yandex.ru: 93.158.134.11 (93.158.134.11, 213.180.193.11, 213.180.204.11, 77.88.21.11, 87.250.250.11, )
Saving in SQL format
Result format:
[% "INSERT INTO dns VALUES('" _ query _ "', '" _ ip _ "', '" _ ips.format('$ip, ') _ "')\n" %]
Example result:
INSERT INTO dns VALUES('vk.com', '87.240.190.67', '87.240.190.67, 87.240.190.72, 87.240.190.78, 93.186.225.208, 87.240.137.158, 87.240.139.194, ')
INSERT INTO dns VALUES('a-parser.com', '176.9.9.90', '176.9.9.90, ')
INSERT INTO dns VALUES('yandex.ru', '5.255.255.50', '5.255.255.50, 77.88.55.50, 77.88.55.88, 5.255.255.5, ')
INSERT INTO dns VALUES('facebook.com', '157.240.19.35', '157.240.19.35, ')
INSERT INTO dns VALUES('youtube.com', '172.217.171.238', '172.217.171.238, ')
Dump results to JSON
Общий формат результата:
[% IF notFirst;
",\n";
ELSE;
notFirst = 1;
END;
obj = {};
obj.query = query;
obj.ips = p1.ips;
obj.json %]
Начальный текст:
[
Конечный текст:
]
Example result:
[
{"ips":[{"ip":"176.9.9.90"}],"query":"http://a-parser.com/"},
{"ips":[{"ip":"172.217.173.78"}],"query":"http://google.com/"},
{"ips":[{"ip":"77.88.55.60"},{"ip":"77.88.55.66"},{"ip":"5.255.255.55"},{"ip":"5.255.255.60"}],"query":"http://yandex.ru/"},
{"ips":[{"ip":"172.217.25.14"}],"query":"http://youtube.com/"},
{"ips":[{"ip":"87.240.190.78"},{"ip":"93.186.225.208"},{"ip":"87.240.137.158"},{"ip":"87.240.139.194"},{"ip":"87.240.190.67"},{"ip":"87.240.190.72"}],"query":"http://vk.com/"},
{"ips":[{"ip":"157.240.196.35"}],"query":"http://facebook.com/"}
]
To make the "Initial Text" and "Final Text" options available in the Task Editor, , you need to activate "More options".
Possible Settings
| Parameter | Default Value | Description |
|---|---|---|
| DNS Servers | 208.67.220.220, 8.8.8.8 | Support for multiple servers separated by comma |
| Not found is error | ☐ | Allows rechecking false responses from DNS servers |
| Rotation type | Consistently | Method for selecting the next DNS server (Sequentially/Randomly) |
| Ban not working servers | ☑ | Ban non-working servers based on a heuristic algorithm for 20 minutes; if only one server is left, it is not banned |
| Query type | A - Address record | Allows selecting the record type |