SecurityTrails::Domain - Data collection about a domain from SecurityTrails

Scraper Overview

SecurityTrails is the world's largest repository of historical DNS data Data for security companies, researchers, and teams that need to detail, find suspicious changes in DNS records, and prevent future fraudulent or criminal activity.

The functionality of A-Parser allows you to save parsing settings for future use (presets), set a parsing schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration over alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in the form and structure that you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Go to DEMOBuy A-Parser Pro ($299)

Collected Data

• Alexa rank
• Apex domain
• Host
• Number of subdomains
• A records
• TXT records
• AAAA records
• Ns records
• SOA records
• MX records
• CNAME records
• CNAME records pointed here
• MX records pointed here
• NS records pointed here
• Subdomains
• historical A
• historical AAAA
• historical MX
• historical NS
• historical SOA
• historical TXT

Capabilities

Choose the Mode section from which data is collected:

Subdomain

• https://securitytrails.com/list/apex_domain/google.com

History

• https://securitytrails.com/domain/google.com/history/a
• https://securitytrails.com/domain/google.com/dns

Use Cases

• Obtaining subdomains and their Rank
• Collecting information about the hosting provider of subdomains
• Obtaining the mail provider of subdomains
• Collecting history about A, AAAA, MX, NS, SOA, TXT
• Collecting domain DNS records A, AAAA, MX, NS, SOA, TXT

Queries

As queries, it is necessary to specify domains, for example:

google.com

Output Results Examples

A-Parser supports flexible formatting of results thanks to the built-in template engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in structured ones, for example CSV or JSON

Default Output

Result format:

$query - $alexa - $apex_domain

Example of result:

google.com - 1 - google.com

Output Results for aRecords

Result format:

$aRecords.format('$ip, $stats\n')

Example of result:

142.250.73.238, 18037

Output Results for txt Records

Result format:

$txt.format('$record\n')

Example of result:

v=spf1 include:_spf.google.com ~all
google-site-verification=wD8N7i1JTNTkezJ49swvWW48f8_9xveREV4oB-0Hf5o
globalsign-smime-dv=CDYX+XFHUw2wml6/Gb8+59BsH31KzUr6c1l2BPvqKX8=
facebook-domain-verification=22rm551cu4k0ab0bxsw536tlds4h95
docusign=1b0a6754-49b1-4db5-8540-d2c12664b289
docusign=05958488-4752-4ef2-95eb-aa7ba8a3bd0e
apple-domain-verification=30afIBcvSuDV2PLX

Output Results for aaaaRecords

Result format:

$aaaaRecords.format('$ip, $stats\n')

Example of result:

2607:f8b0:4004:82a::200e, 17294

Output Results for nsRecords

Result format:

$nsRecords.format('$ns, $stats\n')

Example of result:

ns4.google.com, 5226
ns3.google.com, 5233
ns2.google.com, 5268
ns1.google.com, 5298

Output Results for mxRecords

Result format:

$mxRecords.format('$p, $host, $stats\n')

Example of result:

aspmx.l.google.com, 13227170, 10
alt4.aspmx.l.google.com, 7665791, 50
alt3.aspmx.l.google.com, 7726366, 40
alt2.aspmx.l.google.com, 12894431, 30
alt1.aspmx.l.google.com, 12975832, 20

Output Results for soaRecords

Result format:

$soaRecords.format('$ttl, $email, $stats\n')

Example of result:

dns-admin.google.com, 147485, 900

Output Results for soaRecords in CSV

Result format:

[% FOREACH soaRecords; 
    tools.CSVline(ttl, email, stats);
END %]

Example of result:

dns-admin.google.com,147485,900

Output Information about Subdomains

Result format:

$subdomains.format('$domain, $alexa, $hosting, $mail\n')

Example of result:

google.com, 4, Google LLC, Google LLC
plus.google.com, 11, Google LLC, 
maps.google.com, 18, Google LLC, 
play.google.com, 22, Google LLC, 
docs.google.com, 24, Google LLC, 
support.google.com, 27, Google LLC, 
sites.google.com, 36, Google LLC, 
developers.google.com, 40, Google LLC, 
apis.google.com, 44, Google LLC, 
chrome.google.com, 47, Google LLC, 
translate.google.com, 57, Google LLC

Output Information about Subdomains in CSV Table

Result format:

[% FOREACH subdomains; 
    tools.CSVline(domain, alexa, hosting, mail);
END %]

Example of result:

google.com,4,"Google LLC","Google LLC"
plus.google.com,11,"Google LLC",
maps.google.com,18,"Google LLC",
play.google.com,22,"Google LLC",
docs.google.com,24,"Google LLC",
support.google.com,27,"Google LLC",
sites.google.com,36,"Google LLC",
developers.google.com,40,"Google LLC"

Dump Results to JSON

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.subdomains = [];

FOREACH item IN p1.subdomains;
    obj.subdomains.push({
        domain = item.domain
        alexa = item.alexa
        hosting = item.hosting
        mail = item.mail
    });
END;

obj.json %]

Начальный текст:

[

Конечный текст:

]

Example of result:

[{
    "query": "google.com",
    "subdomains": [
        {
            "domain": "google.com",
            "mail": "Google LLC",
            "hosting": "Google LLC",
            "alexa": 4
        },
        {
            "domain": "plus.google.com",
            "mail": "",
            "hosting": "Google LLC",
            "alexa": 11
        },
        {
            "domain": "maps.google.com",
            "mail": "",
            "hosting": "Google LLC",
            "alexa": 18
        }
    ]
}]

tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Possible Settings

note

Common settings for all scrapers

Parameter	Default Value	Description
Mode	History	The section of the website from which data is collected