Skip to main content

SE::YouTube - fully-featured YouTube Scraper

YouTube

Overview of the scraper​

The YouTube search results scraper. allows you to get large databases of video links, ready for further use. You can use queries in the same form as you enter them in the YouTube search bar. Using the YouTube scraper, you can collect basic data about a video in multithreaded mode. To get full data about each video, you can use SE::YouTube::VideoSE::YouTube::Video

A-Parser's functionality allows you to save YouTube scraper settings for future use (presets), ), set up a scraping schedule, and much more. You can use automatic query multiplication, substitution of subqueries from files, iteration of alphanumeric combinations, and lists to get the maximum possible number of results.

Results can be saved in the form and structure you need, thanks to the built-in powerful templating engine Template Toolkit which allows applying additional logic to the results and outputting data in various formats, including JSON, SQL and CSV.

Collected data​

Data is collected from the service http://www.youtube.com/.

  • Main results
    • Link to the video
    • Video title
    • Video description
    • Username
    • Link to the preview image
    • View count
    • Video duration
    • Date the video was added
    • Channel subscriber count
    • AI description (summary), if available
Collected data
  • Array of "Related queries"
    • Keyword
    • Link to the preview

Capabilities​

  • Maximum number of pages for scraping on YouTube - 50 pages
  • Choice of country from where the search is performed
  • Search by date added
  • Selection of result type ( (videos, channels, playlists)
  • Selection of video duration
  • Advanced search parameter configuration (subtitles, 3D, HD, live, Creative Commons license)
  • Sorting by relevance, date added, rating, view count
  • Additionally scrapes the link to the video preview
  • Option to select the interface language
  • Option to enable safe search mode

Use cases​

  • Search, collection and analysis of information on YouTube

Queries​

As queries, you should specify search phrases, for example:

Football   
Niagara Falls   
Speak in english  
Cats and dogs  
Cars

Query substitutions​

You can use built-in macros to multiply queries. For example, if we want to get a very large database of forums, we can specify several main queries in different languages:

forum
forum
foro

In the query format, we specify character iteration from a to zzzz, . This method allows maximum rotation of search results and retrieval of many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each initial search query, totaling 4 x 475254 = 1901016 search queries. This is an impressive number, but it is not a problem for A-Parser. With a speed of 2000 queries per minute, this task will be processed in just 16 hours.

Examples of output results​

A-Parser offers flexible result formatting thanks to the built-in templating engine Template Toolkit, which allows it to output results in any form, as well as structured formats, such as CSV or JSON

Exporting a list of links​

Same as in SE::Google.

Result format:

[% FOREACH item IN p1.serp;    loop.count _ ' - ' _ item.link _ ' - ' _ item.title _ ' - ' _ item.desc _ "\n"; END %]

Result example:

1 - https://www.youtube.com/watch?v=dm_T7H6J2U8 - HOW SMART ARE YOU? A simple Test that many adults won't pass - In this video, you can check how smart you are. You will have a <b>test</b>, consisting of simple school questions ...
2 - https://www.youtube.com/watch?v=iDAYNEV9Kxg - Unique Japanese test for brain aging! Be sure to check yourself! - Unique Japanese <b>test</b>  for brain aging! Be sure to check yourself! This <b>test</b>  was developed in Japan. How ...
3 - https://www.youtube.com/watch?v=0PEy2_sSy6A - This Simple Test Will Reveal Your Deepest Hidden Fear - Our subconscious is — quite a dark place, requiring many years of psychoanalysis to understand it. And this ...
4 - https://www.youtube.com/watch?v=j6K9nIugzAY - India vs England 2nd Test Day 4 Highlights 2021| Royal Sports Tv - India vs England 2nd <b>Test</b> Day 4 Highlights 2021 India vs England 2nd <b>Test</b> Day 4 Highlights 2021 | ind vs eng <b>test</b> series India vs ...
5 - https://www.youtube.com/watch?v=ALDqwSMVYKQ - PSYCHE TEST/ 929 SECONDS OF LAUGHTER/BEST FAILS OF FEBRUARY 2021 LOL/TRY NOT TO LAUGH! BEST COUB - Telegram channel: https://t.me/CrazyHumor129k ON THE CHANNEL YOU WILL FIND 929SECONDS OF EXCELLENT LAUGHTER WITH THE BEST ...
6 - https://www.youtube.com/watch?v=6X1puBtvc_s - TV Series Pregnancy Test episode 1 - Russian TV series 2015 HD - Series premiere - <b>Pregnancy Test</b>  episode 1 - Russian TV series 2015 After a patient's death, gynecologist Natalya ...
7 - https://www.youtube.com/watch?v=hXuhVD7Dwp0 - Test! Optical Illusions That Will Reveal Unexpected Truth About You! - <b>Test</b>! ! Optical Illusions That Will Reveal Unexpected Truth About You! There are many different types of ...
8 - https://www.youtube.com/watch?v=BYA8lY4o33A - Test! WHAT ANIMAL IS YOUR TALISMAN? What predator is hiding in your soul? Accurate character test - <b>Test</b>! ! WHAT DIVINE ANIMAL ARE YOU? What predator is hiding in your soul? Accurate <b>test</b>  for character To ...
9 - https://www.youtube.com/watch?v=V-kqty2vAm4 - Test! SOMEONE IS SECRETLY IN LOVE WITH YOU! FIND OUT WHO! - <b>Test</b>! ! SOMEONE IS SECRETLY IN LOVE WITH YOU! FIND OUT WHO! You often feel lonely and dream of finding genuine ...
10 - https://www.youtube.com/watch?v=9HtbSe_oJto - Take this Test and check your Attention - In this video, we will check how developed your attention is. You will have different types of tasks with several levels ...
...

The built-in tool tools.CSVLine allows you to create correct tabular documents ready for import into Excel or Google Sheets.

General result format:

[%  FOREACH i IN p1.serp;    tools.CSVline(i.link, i.title, i.desc); END  %]

Filename:

$datefile.format().csv

Initial text:

Link,Anchor,Snippet

tip

In the General result format, the Template Toolkit templating engine is used to output the $serp array in a FOREACH loop.

In the result filename, you just need to change the file extension to csv.

For the "Initial text" option to be available in the Task Editor, , you need to activate "More options". In "Initial text", write the column names separated by commas and make the second line empty.

Keyword competition​

Same as in SE::Google.

Saving in SQL format​

Result format:

[%  FOREACH serp;   "INSERT INTO serp VALUES('" _ query _ "', '";   link _ "', '";  title _ "')\n"; END  %]

Result example:

INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=dm_T7H6J2U8', 'HOW SMART ARE YOU? A Simple Test That Many Adults Won't Pass')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=iDAYNEV9Kxg', 'Unique Japanese test for brain aging! Be sure to check yourself!')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=0PEy2_sSy6A', 'This Simple Test Will Reveal Your Deepest Hidden Fear')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=BYA8lY4o33A', 'Test! WHAT ANIMAL IS YOUR TALISMAN? What predator is hiding in your soul? Accurate character test')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=5Se6w0lOkyY', 'New Renault Duster.Test drive.Anton Avtoman.')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=Ko8cFdoOV6U', 'Test! WHAT KIND OF GIRL ARE YOU? Who is more in you, LADY or TOMBOY?')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=j6K9nIugzAY', 'India vs England 2nd Test Day 4 Highlights 2021| Royal Sports Tv')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=9HtbSe_oJto', 'Take this Test and check your Attention')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=V-kqty2vAm4', 'Test! SOMEONE IS SECRETLY IN LOVE WITH YOU! FIND OUT WHO!')
...

Dump results to JSON​

Same as in SE::Google.

Results processing​

A-Parser allows processing results directly during scraping. In this section, we have provided the most popular use cases for the YouTube scraper

Same as in SE::Google.

Same as in SE::Google.

Extracting domains​

Same as in SE::Google.

Removing tags from video titles and descriptions​

Add Results Builder and select the source from the drop-down list: $p1.serp.$i.title - Title. Select the type: Remove HTML tags.

Add Results Builder again and select the source from the drop-down list: $p1.serp.$i.desc - Description. Select the type: Remove HTML tags.

Download example

How to import the example into A-Parser

eJyVVMtu2zAQ/BWBMJAGUI3k0ItujlsjLZw4tZ1D4fjASCuBDUWyJOXGEPTv3aVo
y07TQ2/kPmb2MWTLPHcv7sGCA+9YtmmZCWeWsdWXLPuhm3XzDMkSar2D5HZ9N088
r1xSWl0na+EluISrIvkMLrfCeKGVYykz3DqwBLg5xUFPASVvpGdpy/zeAPIgsLWi
IKco8G54BbluFMawHZcNxlx3/w4vta25x6ID6iGDjZDfjHvnh4uRp1KTj8mowEKf
ntTFJeu225T1eW4W4ijNXI/jBI7OFd/BWhOVkDCYZ3i753UgK7gH8h4IL8f+lRB4
UQiaCZc9A01kYH1U4lcoVmmMxaMV4GY4WTR5CABk3B+q27BRuDOEaELu9z6HZSWX
DlLmsNQZx0KKtx7hwXKv7SLuKGuZVhMp57ADOYQF/JtGyALXNykx6WtMfD9k8RdG
d2zvlAqX9ttiDUeUcLtZ3A1ZhZ7rCjsvnrFvKWrh8e6mQQkZu0LjC4A5zuyeZlZr
C0eaiBzZUdMGFClkWNnEDKazNs7WcmJsmdONzZFvc5VuGGkKU4OYGMknatKG53Hr
a0k7t5bv0XgIJuX0GaTi9/BIk/8FFxI6TMi1KkW1iE/i0Eaj1viqF2qqayOBhq4a
KVEzDpaDdicuaoQuw/TeJk8DRSCNLxdr0NJ9W/VzNFZgSZ/SWPUpa4TMuZSPy/mp
hw16D1p3BJvjI6o0ypvmHvSfsUrrKjw6eDX4zwAuz9sGui11H3+Z4+/Vnv01Wduh
qn66hz6KuqQYtOG4HEqG/pU/xum9JQ==
tip

You can add the Results Builder as many times as you need.

See also: Results Builder

Same as in SE::Google.

Possible settings​

note

Common settings for all scrapers

Parameter NameDefault ValueDescription
DeviceDesktopSelection of output type (Desktop/Mobile)
Pages count10Number of pages to scrape (from 1 to 50)
Search from countryAuto (Based on IP)Selection of country from where the search is performed
Interface languageEnglishSelection of interface language
Restricted mode☐Enable/disable "Safe search" mode
Uploaded timeAll timeSearch by date added
Result typeVideoSelection of result type
DurationAllSelection of video duration
FeaturesAllAdvanced search parameter configuration
Sort byRelevancySorting of results
Advanced filters (param sp=)Allows specifying complex filter combinations. To do this, you need to take the value of the parameter sp from the URL in the browser and paste it into this field. This value takes priority over the filters defined in the scraper settings.