Skip to main content

Net::Whois - Whois scraper checks domain data, determines whether the domain is registered or not

Overview​

Overview

Data is collected via the WHOIS protocol from central servers for each domain zone. You can read more about WHOIS on Wikipedia: http://ru.wikipedia.org/wiki/WHOIS

A-Parser functionality allows you to save scraping settings for future use (presets), set a scraping schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration over alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the form and structure that you need, thanks to the built-in powerful templating engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Use Cases for the Scraper​

Collected Data​

  • Whether the domain is registered or not
  • Domain registration expiration date
  • When the domain was registered
  • The whois service used
  • List of the domain's name servers
  • The current status of the domain
  • Domain release date

Capabilities​

  • Support for working with international domains (IDN, for example yandex.рф)

Use Cases​

  • Collecting available domains or those that will soon be released
  • Collecting a list of ns-servers
  • Obtaining domain statuses

Queries​

As queries, you need to specify the domain of the site you are searching for, for example:

a-parser.com  
yandex.ru  
google.com  
vk.com  
facebook.com

Output Results Examples​

A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON

Default Output​

Result format:

$query - registered: $registered, expire: $expire_date, creation: $creation_date

The result shows the domain, whether it is registered, the domain registration expiration date, and the domain registration date:

facebook.com - registered: 1, expire: 30.03.2028, creation: 29.03.1997
a-parser.com - registered: 1, expire: 25.02.2022, creation: 25.02.2012
vk.com - registered: 1, expire: 23.06.2021, creation: 24.06.1997
yandex.ru - registered: 1, expire: 01.10.2021, creation: 23.09.1997
google.com - registered: 1, expire: 14.09.2028, creation: 15.09.1997

Output of NS List​

Result format:

$ns.format('$server\n')

Example of result:

demi.ns.cloudflare.com
vern.ns.cloudflare.com

Output of domain terms data in CSV​

Result format:

[% tools.CSVline(query, registered, expire_date, creation_date, free_date, whoisserver, registrar) %]

Example of result:

a-parser.com,1,25.02.2022,25.02.2012,none,whois.crsnic.net,"PDR Ltd. d/b/a PublicDomainRegistry.com"

Output of NS List in CSV Table​

Result format:

[% FOREACH ns;
  tools.CSVline(query, server);
END %]

Example of result:

a-parser.com,demi.ns.cloudflare.com
a-parser.com,vern.ns.cloudflare.com

Dumping Results to JSON​

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.registered = p1.registered;
obj.expire_date = p1.expire_date;
obj.creation_date = p1.creation_date;
obj.free_date = p1.free_date;
obj.whoisserver = p1.whoisserver;
obj.registrar = p1.registrar;
obj.domains = [];

FOREACH item IN p1.ns;
    obj.domains.push({
        server = item.server
    });
END;

obj.json %]

Начальный текст:

[

Конечный текст:

]

Example of the result:

[{
    "registered": 1,
    "query": "a-parser.com",
    "free_date": "none",
    "expire_date": "25.02.2022",
    "registrar": "PDR Ltd. d/b/a PublicDomainRegistry.com",
    "domains": [
        {
            "server": "demi.ns.cloudflare.com"
        },
        {
            "server": "vern.ns.cloudflare.com"
        }
    ],
    "creation_date": "25.02.2012",
    "whoisserver": "whois.crsnic.net"
}]
tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Results processing​

A-Parser allows you to process results directly during scraping, in this section we have provided the most popular cases for the Net::Whois scraper

Getting the domain registrar​

Add Result Constructor and select the source from the dropdown list: $p1.data. Choose the type: RegEx Match. In the Regex field, enter the regular expression: Registrar:\s*(.+) and save to the variable registrar. Add to the Result format: $query - $registrar\n

Download example

How to import an example into A-Parser

eJxtVMlu2zAQ/ZVgkEPSOkJy6EU3x4CBFq6VOg56sH0grJHLhiIZLm4CQf/eIbVQ
SXuyZ3vzhvNGDThmn+2DQYvOQr5rQMf/kIPBE7fOMAMz0MxYNCFhB2t0ef7zl+KW
AiVWzAsHswbcm0YqU2c0hpdIQV6SXSlTM0egMQ3OTPiQdvni0bxd3Fxcjn32ewnt
4TCDLtkuY2XI1XdZT2sMPrIzblXA5wKTe0nWmtWxQ8kchmjWUbi6ztxrQGBlyR1X
komuQxgrdX2S/CUylIpyA0uOdmlUTS6HESBSH9jt+lGAIHys/dHVQF4xYXEGlqgu
GREpP0a4Q8OcMoUOfMjfgJJzIVZ4RpHSIv6956KkHcwrKvraF/4/pfgHox3Hm7ai
Tf0xxGFEidZ98T1VlWqlTsNjCF5zR7ZdKC/DYm7J+Yyoxzdbh7RaGRzb9Mh9dxKa
RhlkkVY218n1box3a5k4G7DKmyP1290SQ+ZYePtefqQmfI3g4TeHzaCufL+3n66y
z9dDcBsLpBeCilVYZFL8oSXEo5IVPxW9ngc6Xm7pZAq5ULUW6EYIb3GTNDi3/a6D
kV7hY/EitiCa4x0RFSXst8fuPbThxOxLYFzTuqZde8gjE+Jps5pGIOmWDHbTHW92
JAmHfIcnRXKlacKQ/WWPX4Bmet9505IGftuHLinMEp10xmTSgiG/a/8CXrN/Jg==

Possible settings​

note

Common settings for all scrapers

ParameterDefault valueDescription
Recursive query☐Allows you to get an extended version of WHOIS
WHOIS serverAllows you to set your own WHOIS server
Checking by DNS☑Allows you to check if a domain is registered based on the existence of an IP for the domain
Checking domains by DNS record☑Checks IP, if it finds status ALLOCATED, then determines the domain as registered, if the option is disabled, then IP check does not occur at all
Force DNS check for not found domains☑Forced check if the domain was determined as unregistered
Last updated