59

I mean, how do they work, generally? How do they receive the link to a video stream itself (not just the page containing a Flash player)?

I did a search on the web but couldn't find anything useful (all links point to such services, but none of them tell how they are actually implemented).

49

There is a very popular open source command-line downloader called youtube-dl, which does exactly that. It grabs the actual video and audio file links from a given YouTube link – or any other popular web video site like Vimeo, Yahoo! Video, uStream, etc.

To see how that's done, look into the YouTube extractor. That's just too much to show here. Other extractors exist for simpler sites. Steven Penny has a simple JavaScript downloader for YouTube too, which is a little more straightforward.

But basically, for a Flash video player, it must be initialized and configured through some JavaScript. Simply speaking, the Flash object's player will receive a URL of a video stream to load.

In order to find the video stream, you'd have to parse the HTML and JS code of the video page to find the relevant initialization code, and then from there try to find the link to the actual MP4 file. It might be there in plaintext, but it could also be generated on the fly with some specific download tokens. Often, the JavaScript is obfuscated to make it harder to re-engineer it. Or the video information might be contained in an XML file that's loaded asynchronously by JS.

For HTML5 progressive download video, the actual source file is usually mentioned directly in the source child of the video tag, so if you'd search the page for mp4 or similar. For example on German news show Tagesschau 100, you'll find:

<source src="http://media.tagesschau.de/video/2014/0626/TV-20140626-1649-5801.webl.h264.mp4" type="video/mp4">

For more advanced playback technologies like MPEG DASH or Apple's HTTP Live Streaming (HLS), you have to parse a meta-information file to get the actual video stream. The meta file (.mpd for example in DASH, and .m3u8 for HLS) will contain links to segments of video and audio, which you'd later have to combine to get a playable file.

There's no general solution for this. It requires careful inspection and debugging of the target site.

  • 3
    One question, what is Youtube/Google's policy on this? Are they ok with this, or not so much? – JMK Jun 26 '14 at 15:53
  • 31
    The YouTube Terms of Service in §5.1.L disallow consumption of their content through any other means than streaming, so theoretically it's not allowed. In practice, they won't be able to enforce that though. Any downloader can more or less simulate that it's just streaming. – slhck Jun 26 '14 at 16:23
  • 2
    @StevenPenny do you have any non minified version of that? – TankorSmash Jun 26 '14 at 18:08
  • 5
    @slhck Flash could also make HTTP requests by itself. Instead, it uses the browser’s HTTP engine. If Flash itself made the requests, they wouldn’t be “visible” to the browser. Sure would be great for advertisers. ;) – Daniel B Jun 26 '14 at 19:49
  • 3
    @slhck they can't enforce it programmatically, but if they got their lawyer team out could they enforce it legally? – Cruncher Jun 27 '14 at 15:45
32

YouTube Bookmarklet

This is how I did it with JavaScript

Start with ytplayer.config.args object. This contains all URLs for the video. It is broken up into

url_encoded_fmt_stream_map // traditional: contains video and audio stream
adaptive_fmts              // DASH: contains video or audio stream

Each of these is a comma separated array of what I would call "stream objects". Each "stream object" will contain values like this

url  // direct HTTP link to a video
itag // code specifying the quality
s    // signature, security measure to counter downloading

Each URL will be encoded so you will need to decode them. Now the tricky part.

YouTube has at least 3 security levels for their videos

unsecured // as expected, you can download these with just the unencoded URL
s         // see below
RTMPE     // uses "rtmpe://" protocol, no known method for these

The RTMPE videos are typically used on official full length movies, and are protected with SWF Verification Type 2. This has been around since 2011 and has yet to be reverse engineered.

The type "s" videos are the most difficult that can actually be downloaded. You will typcially see these on VEVO videos and the like. They start with a signature such as

AA5D05FA7771AD4868BA4C977C3DEAAC620DE020E.0F421820F42978A1F8EAFCDAC4EF507DB5

Then the signature is scrambled with a function like this

function mo(a) {
  a = a.split("");
  a = lo.rw(a, 1);
  a = lo.rw(a, 32);
  a = lo.IC(a, 1);
  a = lo.wS(a, 77);
  a = lo.IC(a, 3);
  a = lo.wS(a, 77);
  a = lo.IC(a, 3);
  a = lo.wS(a, 44);
  return a.join("")
}

This function is dynamic, it typically changes every day. To make it more difficult the function is hosted at a URL such as

http://s.ytimg.com/yts/jsbin/html5player-en_US-vflycBCEX.js

this introduces the problem of Same-origin policy. Essentially, you cannot download this file from www.youtube.com because they are different domains. A workaround of this problem is CORS. With CORS, s.ytimg.com could add this header

Access-Control-Allow-Origin: http://www.youtube.com

and it would allow the JavaScript to download from www.youtube.com. Of course they do not do this. A workaround for this workaround is to use a CORS proxy. This is a proxy that responds with the following header to all requests

Access-Control-Allow-Origin: *

So, now that you have proxied your JS file, and used the function to scramble the signature, you can use that in the querystring to download a video.

  • 1
    Do you happen to know (for DASH playback) how the YouTube player determines the byte ranges to be requested from the media representation? The MPD file only lists segments. – slhck Jul 9 '14 at 12:51
  • 2
    Not exactly.. There is a single file for video and audio, but YouTube requests them by byte ranges, in several chunks. If you switch to another quality, it'll change the byte range too. Just wondering how the player knows which second corresponds to which byte offset. – slhck Jul 9 '14 at 16:25
  • 1
    Can you give an example of an s type video and RTMPE type video? – Chloe May 3 '16 at 5:10
  • @SurajJain here is new page - I will be rewriting this answer as I use a different method now github.com/svnpenn/umber/blob/master/bmklet/youtube/download.js – Steven Penny Feb 19 '18 at 12:39
0

My answer: from 22 January 2019, using these methods can get caught if you try to bypass without linking your user information as well.

Why? since I'm a new user to this platform, I cannot comment for rule specified by @Daniel-B. According to new ToS (in German as I am in Germany; please translate) for YouTube under $6.1 G$:

You agree any automated system (including – but not limited to – any robot, spider or offline reader) to use that on the website accesses in such a way that more requests to the server within a specified time YouTube directed being able to reasonably produce as a human within the same time period using a publicly available, unmodified standard web browser;

Now they can find out the time duration for each request and can track if you are violating. How is it possible now, given this scenario and your external IP address will be known even if you use a VPN to protect yourself without linking details of user to any service.

  • 1
    Welcome traveler from the future...It isn’t clear how this answers the author’s question – Ramhound Dec 27 '18 at 22:31
  • If you wanted to warn the author, with regards to a specific answer, you should have submitted a comment and downvoted any answer you didn't think was helpful due to the possible legal issues that could follow. This answer reads more like what you would find on a discussion forum post, and Super User, isn't a discussion forum – Ramhound Jan 13 at 0:34

protected by Ramhound Dec 27 '18 at 22:31

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.