Skip to main content

SE::Yandex::SafeBrowsing - Checking a domain in the Yandex blacklist

Overview of the scraper​

Overview of the scraperSE::Yandex::SafeBrowsingSE::Yandex::SafeBrowsing – checks for the presence of a domain in the Yandex blacklist.

A-Parser functionality allows you to save scraping settings for the SE::Yandex::SafeBrowsing scraper for later use (presets), ), set a scraping schedule, and much more.

Saving results is possible in the form and structure you need, thanks to the integrated powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.

Collected Data​

  • Presence in the suspicious website list

what data the SE::Yandex::SafeBrowsing scraper collects

Use Cases​

  • Checking a list of domains for inclusion in the Yandex blacklist
  • Monitoring your domains for inclusion in the Yandex blacklist

Queries​

The search website URL must be specified as queries, for example:

http://a-parser.com/  
http://www.yandex.ru/  
http://google.com/  
http://vk.com/  
http://facebook.com/  
http://youtube.com/

Output Results Examples​

A-Parser supports flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in an arbitrary form, as well as in a structured format, such as CSV or JSON

Default Output​

Result Format:

$query: $exists\n

Example result showing the URL and whether it is in the Yandex blacklist:

http://vk.com/: 0  
http://www.yandex.ru/: 0  
http://youtube.com/: 0  
http://facebook.com/: 0  
http://google.com/: 0  
http://a-parser.com/: 0

Saving in SQL Format​

Result Format:

[% "INSERT INTO yandexSafeBrowsing VALUES('" _ query _ "', '" _ p1.exists _ "')\n" %]

Example Result:

INSERT INTO dns VALUES('http://google.com', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://yandex.ru', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://a-parser.com', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://vk.com', '0')
INSERT INTO yandexSafeBrowsing VALUES('http://facebook.com', '0')

Dump Results to JSON​

Общий формат результата:

[% IF notFirst;
  ",\n";
ELSE;
  notFirst = 1;
END;

obj = {};
obj.query = query;
obj.ip = ip;

obj.json %]

Начальный текст:

[

Конечный текст:

]

Example Result:

[{"ip":"31.13.79.35","query":"facebook.com"},
{"ip":"172.217.174.174","query":"google.com"},
{"ip":"5.255.255.50","query":"yandex.ru"},
{"ip":"142.250.71.14","query":"youtube.com"},
{"ip":"87.240.190.67","query":"vk.com"},
{"ip":"176.9.9.90","query":"a-parser.com"}]
tip

To make the "Initial Text" and "Final Text" options available in the Task Editor, , you need to activate "More Options".

Possible Settings​

note

General settings for all scrapers