Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I mean, how do they work, generally? How do they receive the link to a video stream itself (not just the page containing a Flash player)?

I did a search on the web but couldn't find anything useful (all links point to such services, but none of them tell how they are actually implemented).

share|improve this question
add comment

2 Answers 2

up vote 29 down vote accepted

There is a very popular open source command-line downloader called youtube-dl, which does exactly that. It grabs the actual video and audio file links from a given YouTube link – or any other popular web video site like Vimeo, Yahoo! Video, uStream, etc.

To see how that's done, look into the YouTube extractor. That's just too much to show here. Other extractors exist for simpler sites. Steven Penny has a simple JavaScript downloader for YouTube too, which is a little more straightforward.

But basically, for a Flash video player, it must be initialized and configured through some JavaScript. Simply speaking, the Flash object's player will receive a URL of a video stream to load.

In order to find the video stream, you'd have to parse the HTML and JS code of the video page to find the relevant initialization code, and then from there try to find the link to the actual MP4 file. It might be there in plaintext, but it could also be generated on the fly with some specific download tokens. Often, the JavaScript is obfuscated to make it harder to re-engineer it. Or the video information might be contained in an XML file that's loaded asynchronously by JS.

For HTML5 progressive download video, the actual source file is usually mentioned directly in the source child of the video tag, so if you'd search the page for mp4 or similar. For example on German news show Tagesschau 100, you'll find:

<source src="http://media.tagesschau.de/video/2014/0626/TV-20140626-1649-5801.webl.h264.mp4" type="video/mp4">

For more advanced playback technologies like MPEG DASH or Apple's HTTP Live Streaming (HLS), you have to parse a meta-information file to get the actual video stream. The meta file (.mpd for example in DASH, and .m3u8 for HLS) will contain links to segments of video and audio, which you'd later have to combine to get a playable file.

There's no general solution for this. It requires careful inspection and debugging of the target site.

share|improve this answer
    
One question, what is Youtube/Google's policy on this? Are they ok with this, or not so much? –  JMK Jun 26 at 15:53
20  
The YouTube Terms of Service in §5.1.L disallow consumption of their content through any other means than streaming, so theoretically it's not allowed. In practice, they won't be able to enforce that though. Any downloader can more or less simulate that it's just streaming. –  slhck Jun 26 at 16:23
    
Well, the URL need not come from an external source, a Flash application could generate it itself. Still, because (for whatever reason) all actual HTTP requests are deferred to the hosting browser, the URLs are easily found out. –  Daniel B Jun 26 at 17:41
1  
@StevenPenny do you have any non minified version of that? –  TankorSmash Jun 26 at 18:08
2  
@slhck Flash could also make HTTP requests by itself. Instead, it uses the browser’s HTTP engine. If Flash itself made the requests, they wouldn’t be “visible” to the browser. Sure would be great for advertisers. ;) –  Daniel B Jun 26 at 19:49
show 3 more comments

YouTube Bookmarklet

This is how I did it with JavaScript

Start with ytplayer.config.args object. This contains all URLs for the video. It is broken up into

url_encoded_fmt_stream_map // traditional: contains video and audio stream
adaptive_fmts              // DASH: contains video or audio stream

Each of these is a comma separated array of what I would call "stream objects". Each "stream object" will contain values like this

url  // direct HTTP link to a video
itag // code specifying the quality
sig  // signature, security measure to counter downloading
s    // different type of signature

Each URL will be encoded so you will need to decode them. Now the tricky part.

YouTube has at least 4 security levels for their videos

unsecured // as expected, you can download these with just the unencoded URL
sig       // these require the "sig" value in the querystring as mentioned
s         // see below
RTMPE     // uses "rtmpe://" protocol, no known method for these

The RTMPE videos are typically used on official full length movies, and are protected with SWF Verification Type 2. This has been around almost 3 years and has yet to be reverse engineered.

The type "s" videos are the most difficult that can actually be downloaded. You will typcially see these on VEVO videos and the like. They start with a signature such as

AA5D05FA7771AD4868BA4C977C3DEAAC620DE020E.0F421820F42978A1F8EAFCDAC4EF507DB5

Then the signature is scrambled with a function like this

function mo(a) {
  a = a.split("");
  a = lo.rw(a, 1);
  a = lo.rw(a, 32);
  a = lo.IC(a, 1);
  a = lo.wS(a, 77);
  a = lo.IC(a, 3);
  a = lo.wS(a, 77);
  a = lo.IC(a, 3);
  a = lo.wS(a, 44);
  return a.join("")
}

This function is dynamic, it typically changes every day. To make it more difficult the function is hosted at a URL such as

http://s.ytimg.com/yts/jsbin/html5player-en_US-vflycBCEX.js

this introduces the problem of Same-origin policy. Essentially, you cannot download this file from www.youtube.com because they are different domains. A workaround of this problem is CORS. With CORS, s.ytimg.com could add this header

Access-Control-Allow-Origin: http://www.youtube.com

and it would allow the JavaScript to download from www.youtube.com. Of course they do not do this. A workaround for this workaround is to use a CORS proxy. This is a proxy that responds with the following header to all requests

Access-Control-Allow-Origin: *

So, now that you have proxied your JS file, and used the function to scramble the signature, you can use that in the querystring to download a video.

share|improve this answer
    
Do you happen to know (for DASH playback) how the YouTube player determines the byte ranges to be requested from the media representation? The MPD file only lists segments. –  slhck Jul 9 at 12:51
    
@slhck YouTube DASH uses a single file for video stream and a single file for audio stream, for a total of 2 files. Not sure if that is what you are asking –  Steven Penny Jul 9 at 15:14
    
Not exactly.. There is a single file for video and audio, but YouTube requests them by byte ranges, in several chunks. If you switch to another quality, it'll change the byte range too. Just wondering how the player knows which second corresponds to which byte offset. –  slhck Jul 9 at 16:25
    
@slhck I made the script so I could have a local copy of the videos. I am not sure about your question, but github.com/K-S-V might know –  Steven Penny Jul 9 at 17:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.