Skip to main content

SE::Yandex::SafeBrowsing - Checking domain in Yandex blacklist

Overview​

OverviewSE::Yandex::SafeBrowsingSE::Yandex::SafeBrowsing – checks if a domain is listed in the Yandex blacklist.

A-Parser functionality allows you to save scraper settings SE::Yandex::SafeBrowsing for future use (presets), schedule scraping tasks, and much more.

Saving results is possible in the form and structure you need, thanks to the built-in powerful template engine Template Toolkit which allows you to apply additional logic to the results and output data in various formats, including JSON, SQL, and CSV.

Collected data​

  • Presence in the list of suspicious sites

what data does the scraper SE::Yandex::SafeBrowsing collect

Use cases​

  • Checking a list of domains for inclusion in the Yandex blacklist
  • Monitoring your own domains for inclusion in the Yandex blacklist

Queries​

As queries, you need to specify the URL of the site you are looking for, for example:

http://a-parser.com/  
http://www.yandex.ru/  
http://google.com/  
http://vk.com/  
http://facebook.com/  
http://youtube.com/

Output results examples​

A-Parser supports flexible results formatting thanks to the built-in template engine Template Toolkit, which allows it to output results in any form, as well as in structured formats, such as CSV or JSON

Default output​

Result format:

$query: $exists\n

Example of a result showing the URL and whether it is on the Yandex blacklist:

http://vk.com/: 0  
http://www.yandex.ru/: 0  
http://youtube.com/: 0  
http://facebook.com/: 0  
http://google.com/: 0  
http://a-parser.com/: 0

Saving in SQL format​

Result format:

[% "INSERT INTO yandexSafeBrowsing VALUES('" _ query _ "', '" _ p1.exists _ "')\n" %]

Example of a result:

INSERT INTO dns VALUES('http://google.com', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://yandex.ru', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://a-parser.com', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://vk.com', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://facebook.com', '0')

Dumping results to JSON​

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.ip = ip;

obj.json %]

Начальный текст:

[

Конечный текст:

]

Example of a result:

[{"ip":"31.13.79.35","query":"facebook.com"},
{"ip":"172.217.174.174","query":"google.com"},
{"ip":"5.255.255.50","query":"yandex.ru"},
{"ip":"142.250.71.14","query":"youtube.com"},
{"ip":"87.240.190.67","query":"vk.com"},
{"ip":"176.9.9.90","query":"a-parser.com"}]
tip

To make the "Start text" and "End text" options available in the Task Editor, you need to activate "More options".

Possible settings​

note

Common settings for all scrapers

Last updated