Skip to main content

Net::Whois - Whois Scraper checks domain data, determines if a domain is registered or not

Scraper Overview​

Overview

Data is collected via the WHOIS protocol from central servers for each domain zone. You can read more about WHOIS on Wikipedia: http://ru.wikipedia.org/wiki/WHOIS

A-Parser functionality allows saving scraping settings for future use (presets), scheduling scraping, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration through alphanumeric combinations and lists to get the maximum possible number of results.

Results can be saved in the form and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.

Scraper Use Cases​

Collected Data​

  • Whether the domain is registered or not
  • Domain registration expiration date
  • When the domain was registered
  • Used whois service
  • Domain name server list
  • Current domain status
  • Domain release date

Capabilities​

  • Support for international domains (IDN, e.g., яндекс.рф)

Use Cases​

  • Collecting free domains or those that will be released soon
  • Collecting a list of NS servers
  • Obtaining domain statuses

Queries​

Queries should specify the domain of the sought website, for example:

a-parser.com  
yandex.ru  
google.com  
vk.com  
facebook.com

Output Results Examples​

A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in arbitrary form, as well as in structured formats, such as CSV or JSON

Default Output​

Result format:

$query - registered: $registered, expire: $expire_date, creation: $creation_date

The result displays the domain, whether it is registered, the expiration date, and the domain registration date:

facebook.com - registered: 1, expire: 30.03.2028, creation: 29.03.1997
a-parser.com - registered: 1, expire: 25.02.2022, creation: 25.02.2012
vk.com - registered: 1, expire: 23.06.2021, creation: 24.06.1997
yandex.ru - registered: 1, expire: 01.10.2021, creation: 23.09.1997
google.com - registered: 1, expire: 14.09.2028, creation: 15.09.1997

Outputting NS list​

Result format:

$ns.format('$server\n')

Result example:

demi.ns.cloudflare.com
vern.ns.cloudflare.com

Outputting domain period data in CSV​

Result format:

[% tools.CSVline(query, registered, expire_date, creation_date, free_date, whoisserver, registrar) %]

Result example:

a-parser.com,1,25.02.2022,25.02.2012,none,whois.crsnic.net,"PDR Ltd. d/b/a PublicDomainRegistry.com"

Outputting NS list to CSV table​

Result format:

[% FOREACH ns;
  tools.CSVline(query, server);
END %]

Result example:

a-parser.com,demi.ns.cloudflare.com
a-parser.com,vern.ns.cloudflare.com

Dumping results to JSON​

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.registered = p1.registered;
obj.expire_date = p1.expire_date;
obj.creation_date = p1.creation_date;
obj.free_date = p1.free_date;
obj.whoisserver = p1.whoisserver;
obj.registrar = p1.registrar;
obj.domains = [];

FOREACH item IN p1.ns;
    obj.domains.push({
        server = item.server
    });
END;

obj.json %]

Начальный текст:

[

Конечный текст:

]

Result example:

[{
    "registered": 1,
    "query": "a-parser.com",
    "free_date": "none",
    "expire_date": "25.02.2022",
    "registrar": "PDR Ltd. d/b/a PublicDomainRegistry.com",
    "domains": [
        {
            "server": "demi.ns.cloudflare.com"
        },
        {
            "server": "vern.ns.cloudflare.com"
        }
    ],
    "creation_date": "25.02.2012",
    "whoisserver": "whois.crsnic.net"
}]
tip

For the "Initial text" and "Final text" options to be available in the Job Editor, you need to activate "More options".

Results Processing​

A-Parser allows processing results directly during scraping, in this section we have provided the most popular use cases for the Net::Whois scraper

Getting the domain registrar​

Add Results Constructor and select the source from the dropdown list: $p1.data. Select type: RegEx Match. In the Regex field, enter the regular expression: Registrar:\s*(.+) and save to the variable registrar. Add to Result Format: $query - $registrar\n

Download example

How to import a preset into A-Parser

eJxtVMlu2zAQ/ZVgkEPSOkJy6EU3x4CBFq6VOg56sH0grJHLhiIZLm4CQf/eIbVQ
SXuyZ3vzhvNGDThmn+2DQYvOQr5rQMf/kIPBE7fOMAMz0MxYNCFhB2t0ef7zl+KW
AiVWzAsHswbcm0YqU2c0hpdIQV6SXSlTM0egMQ3OTPiQdvni0bxd3Fxcjn32ewnt
4TCDLtkuY2XI1XdZT2sMPrIzblXA5wKTe0nWmtWxQ8kchmjWUbi6ztxrQGBlyR1X
komuQxgrdX2S/CUylIpyA0uOdmlUTS6HESBSH9jt+lGAIHys/dHVQF4xYXEGlqgu
GREpP0a4Q8OcMoUOfMjfgJJzIVZ4RpHSIv6956KkHcwrKvraF/4/pfgHox3Hm7ai
Tf0xxGFEidZ98T1VlWqlTsNjCF5zR7ZdKC/DYm7J+Yyoxzdbh7RaGRzb9Mh9dxKa
RhlkkVY218n1box3a5k4G7DKmyP1290SQ+ZYePtefqQmfI3g4TeHzaCufL+3n66y
z9dDcBsLpBeCilVYZFL8oSXEo5IVPxW9ngc6Xm7pZAq5ULUW6EYIb3GTNDi3/a6D
kV7hY/EitiCa4x0RFSXst8fuPbThxOxLYFzTuqZde8gjE+Jps5pGIOmWDHbTHW92
JAmHfIcnRXKlacKQ/WWPX4Bmet9505IGftuHLinMEp10xmTSgiG/a/8CXrN/Jg==

Possible Settings​

note

Common settings for all scrapers

ParameterDefault ValueDescription
Recursive query☐Allows obtaining an extended WHOIS version
WHOIS serverAllows setting your own WHOIS server
Checking by DNS☑Allows checking if the domain is registered based on the domain's IP address existence
Checking domains by DNS record☑IP check: if status ALLOCATED is found, the domain is determined as registered; if the option is disabled, no IP check occurs at all
Force DNS check for not found domains☑Forced check if the domain was determined to be unregistered