Skip to main content

SE::YouTube - full-featured YouTube scraper

YouTube

Scraper Overview​

YouTube search results scraper. Thanks to the YouTube scraper, you can obtain large databases of video links ready for further use. You can use queries in the same form as you enter them into the YouTube search bar. Using the YouTube scraper, you can collect basic data for a video clip in multithreaded mode. To obtain full data for each video, you can use SE::YouTube::VideoSE::YouTube::Video

A-Parser functionality allows you to save YouTube scraper settings for future use (presets), set scraping schedules, and much more. You can use automatic query multiplication, substitution of subqueries from files, and iteration through alphanumeric combinations and lists to obtain the maximum possible number of results.

Saving results is possible in any form and structure you need, thanks to the powerful built-in Template Toolkit templating engine, which allows you to apply additional logic to results and output data in various formats, including JSON, SQL, and CSV.

Collected Data​

Data is collected from the http://www.youtube.com/ service.

  • Main search results
    • Video link
    • Video title
    • Video description
    • Username
    • Preview image link
    • Number of views
    • Video length
    • Video upload date
    • Channel subscriber count
    • AI summary, if available
Collected data
  • "Related searches" array
    • Keyword
    • Preview link

Features​

  • Maximum number of pages for scraping in YouTube - 50 pages
  • Country selection from which the search is performed
  • Search by upload date
  • Selection of result type (videos, channels, playlists)
  • Selection of video duration
  • Advanced search settings (subtitles, 3D, HD, live stream, Creative Commons license)
  • Sort by relevance, upload date, rating, view count
  • Additionally scrapes the video preview link
  • Ability to choose the interface language
  • Ability to enable SafeSearch mode

Use Cases​

  • Searching, collecting, and analyzing information on YouTube

Queries​

You should specify search phrases as queries, for example:

Football   
Niagara Falls   
Speak in english  
Cats and dogs  
Cars

Query Substitutions​

You can use built-in macros for query multiplication; for example, if we want to get a very large database of forums, we specify several main queries in different languages:

forum
forum
foro

In the query format, we specify character iteration from a to zzzz; this method allows for maximum rotation of search results and obtaining many new unique results:

$query {az:a:zzzz}

This macro will create 475254 additional queries for each original search query, which in total gives 4 x 475254 = 1901016 search queries—an impressive figure, but no problem at all for A-Parser. At a speed of 2000 queries per minute, such a task will be processed in just 16 hours.

Output Results Examples​

A-Parser supports flexible result formatting thanks to the built-in Template Toolkit templating engine, which allows it to output results in any form, as well as in structured formats like CSV or JSON.

Exporting a List of Links​

Same as in SE::Google.

Result format:

[% FOREACH item IN p1.serp;    loop.count _ ' - ' _ item.link _ ' - ' _ item.title _ ' - ' _ item.desc _ "\n"; END %]

Result example:

1 - https://www.youtube.com/watch?v=dm_T7H6J2U8 - HOW SMART ARE YOU? A Simple Test that many adults won't pass - In this video you can check how smart you are. A <b>test</b>, awaits you, consisting of simple school questions ...
2 - https://www.youtube.com/watch?v=iDAYNEV9Kxg - Unique Japanese brain aging test! Be sure to check yourself! - Unique Japanese <b>test</b> for brain aging! Be sure to check yourself! This <b>test</b> was developed in Japan. How ...
3 - https://www.youtube.com/watch?v=0PEy2_sSy6A - This Simple Test Will Reveal Your Deepest Fear - Our subconscious is a rather dark place; understanding it takes years of psychoanalysis. And this...
4 - https://www.youtube.com/watch?v=j6K9nIugzAY - India vs England 2nd Test Day 4 Highlights 2021| Royal Sports Tv - India vs England 2nd <b>Test</b> Day 4 Highlights 2021 India vs England 2nd <b>Test</b> Day 4 Highlights 2021 | ind vs eng <b>test</b> series India vs ...
5 - https://www.youtube.com/watch?v=ALDqwSMVYKQ - PSYCHOLOGY TEST/ 929 SECONDS OF LAUGHTER/BEST JOKES FOR FEBRUARY 2021 LOL/TRY NOT TO LAUGH! BEST COUB - Telegram channel: https://t.me/CrazyHumor129k ON THE CHANNEL YOU WILL FIND 929 SECONDS OF EXCELLENT LAUGHTER TO THE BEST...
6 - https://www.youtube.com/watch?v=6X1puBtvc_s - Pregnancy Test Series Episode 1 - Russian series 2015 HD - Series Premiere - <b>Test</b> for pregnancy Episode 1 - Russian series 2015 After the death of a patient, gynecologist Natalya ...
7 - https://www.youtube.com/watch?v=hXuhVD7Dwp0 - Test! Optical Illusions That Will Reveal Unexpected Truths About You! - <b>Test</b>! Optical Illusions That Will Reveal Unexpected Truths About You! There are many different types ...
8 - https://www.youtube.com/watch?v=BYA8lY4o33A - Test! WHICH ANIMAL IS YOUR TALISMAN? Which predator hides in your soul? Accurate character test - <b>Test</b>! WHICH DIVINE ANIMAL ARE YOU? Which predator hides in your soul? Accurate <b>test</b> for character In order ...
9 - https://www.youtube.com/watch?v=V-kqty2vAm4 - Test! SOMEONE IS SECRETLY IN LOVE WITH YOU! FIND OUT WHO! - <b>Test</b>! SOMEONE IS SECRETLY IN LOVE WITH YOU! FIND OUT WHO! You often feel lonely and dream of finding true ...
10 - https://www.youtube.com/watch?v=9HtbSe_oJto - Take this Test and check your Attention - In this video we will check how developed your attention is. Different types of tasks with several levels await you...
...

The built-in tools.CSVLine tool allows you to create correct tabular documents ready for import into Excel or Google Sheets.

General result format:

[%  FOREACH i IN p1.serp;    tools.CSVline(i.link, i.title, i.desc); END  %]

File name:

$datefile.format().csv

Initial text:

Link,Anchor,Snippet

tip

In the Result Format, the Template Toolkit templating engine is used to output the $serp array in a FOREACH loop.

In the result file name, you simply need to change the file extension to csv.

To make the "Prepend text" option available in the Task Editor, you need to activate "More options". In "Prepend text", write the column names separated by commas and make the second line empty.

Keyword Competition​

Same as in SE::Google.

Saving in SQL Format​

Result format:

[%  FOREACH serp;   "INSERT INTO serp VALUES('" _ query _ "', '";   link _ "', '";  title _ "')\n"; END  %]

Result example:

INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=dm_T7H6J2U8', 'HOW SMART ARE YOU? A Simple Test that many adults won't pass')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=iDAYNEV9Kxg', 'Unique Japanese brain aging test! Be sure to check yourself!')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=0PEy2_sSy6A', 'This Simple Test Will Reveal Your Deepest Fear')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=BYA8lY4o33A', 'Test! WHICH ANIMAL IS YOUR TALISMAN? Which predator hides in your soul? Accurate character test')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=5Se6w0lOkyY', 'New Renault Duster. Test drive. Anton Avtoman.')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=Ko8cFdoOV6U', 'Test! WHAT KIND OF GIRL ARE YOU? Who is more in you: LADY or TOMBOY?')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=j6K9nIugzAY', 'India vs England 2nd Test Day 4 Highlights 2021| Royal Sports Tv')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=9HtbSe_oJto', 'Take this Test and check your Attention')
INSERT INTO serp VALUES('test', 'https://www.youtube.com/watch?v=V-kqty2vAm4', 'Test! SOMEONE IS SECRETLY IN LOVE WITH YOU! FIND OUT WHO!')
...

Dumping Results to JSON​

Same as in SE::Google.

Results Processing​

A-Parser allows you to process results directly during scraping; in this section, we have listed the most popular use cases for the YouTube scraper.

Same as in SE::Google.

Same as in SE::Google.

Extracting Domains​

Same as in SE::Google.

Removing Tags from Video Titles and Descriptions​

Add a Result Builders and select the source from the dropdown list: $p1.serp.$i.title - Title. Select type: Remove HTML tags.

Add another Result Builders and select the source from the dropdown list: $p1.serp.$i.desc - Description. Select type: Remove HTML tags.

Download example

How to import an example into A-Parser

eJyVVMtu2zAQ/BWBMJAGUI3k0ItujlsjLZw4tZ1D4fjASCuBDUWyJOXGEPTv3aVo
y07TQ2/kPmb2MWTLPHcv7sGCA+9YtmmZCWeWsdWXLPuhm3XzDMkSar2D5HZ9N088
r1xSWl0na+EluISrIvkMLrfCeKGVYykz3DqwBLg5xUFPASVvpGdpy/zeAPIgsLWi
IKco8G54BbluFMawHZcNxlx3/w4vta25x6ID6iGDjZDfjHvnh4uRp1KTj8mowEKf
ntTFJeu225T1eW4W4ijNXI/jBI7OFd/BWhOVkDCYZ3i753UgK7gH8h4IL8f+lRB4
UQiaCZc9A01kYH1U4lcoVmmMxaMV4GY4WTR5CABk3B+q27BRuDOEaELu9z6HZSWX
DlLmsNQZx0KKtx7hwXKv7SLuKGuZVhMp57ADOYQF/JtGyALXNykx6WtMfD9k8RdG
d2zvlAqX9ttiDUeUcLtZ3A1ZhZ7rCjsvnrFvKWrh8e6mQQkZu0LjC4A5zuyeZlZr
C0eaiBzZUdMGFClkWNnEDKazNs7WcmJsmdONzZFvc5VuGGkKU4OYGMknatKG53Hr
a0k7t5bv0XgIJuX0GaTi9/BIk/8FFxI6TMi1KkW1iE/i0Eaj1viqF2qqayOBhq4a
KVEzDpaDdicuaoQuw/TeJk8DRSCNLxdr0NJ9W/VzNFZgSZ/SWPUpa4TMuZSPy/mp
hw16D1p3BJvjI6o0ypvmHvSfsUrrKjw6eDX4zwAuz9sGui11H3+Z4+/Vnv01Wduh
qn66hz6KuqQYtOG4HEqG/pU/xum9JQ==
tip

You can add as many Results Builders as you need.

See also: Results Builder

Same as in SE::Google.

Possible Settings​

note

General settings for all scrapers

Parameter nameDefault valueDescription
DeviceDesktopSelection of output type (Desktop/Mobile)
Pages count10Number of pages to scrape (from 1 to 50)
Search from countryAuto (Based on IP)Selection of the country from which the search is performed
Interface languageEnglishSelection of interface language
Restricted mode☐Enable/disable search "SafeSearch mode"
Uploaded timeAll timeSearch by upload date
Result typeVideoSelection of result type
DurationAllSelection of video duration
FeaturesAllAdvanced search parameter settings
Sort byRelevancySorting of results
Advanced filters (param sp=)Allows specifying complex filter combinations. To do this, take the value of the sp parameter from the URL in the browser and paste it into this field. This value takes priority over the filters set in the scraper settings.