Edit: the things I've written below hold for HTML5 version (in which browser controls everything), in Flash version, Flash settings may be important as well.
Unfortunately, regarding particularly YouTube, the case is very complicated (perhaps it's more simple on other video sites).
First, ad Firefox caching audio/video files, these
about:config setting may affect things:
You need to have at least one of two kinds of caches enabled, and appropriate cache size set, plus additionally high enough
max_entry_size is not very high, this makes sense, as you don't want generally to wipe half of your cache in order to store a HD VEVO video.
Ok, so browser side is fine. Next step is server-side cache restrictions. I've opened random YouTube video (user uploaded, not a copyrighted stuff, it may differ but haven't checked), and here are the response headers of the FLV file (taken with Fiddler):
Cache-control: private means file can be cached in your browser, but not by any intermediary caches (e.g. ISP cache)
max-age are both specified,
14.9.3 Modifications of the Basic Expiration Mechanism
If a response includes both an Expires header and a max-age directive, the max-age directive overrides the Expires header, even if the Expires header is more restrictive
So far so good,
max-age means we are allowed cache it locally in the browser for ~6hours.
But let's refresh the page or load it in the new tab and compare the list of HTTP requests:
Seems that YouTube adds params either to track of numbers of plays, to fight abuse (to have non-predictable video URLs) or whatever.
When opening in a new tab, I've even sometimes seen different servers targeted (load balancer at work):
It's the same with Flash-based and HTML5-based video watching.
Because URLs are different (even if this was by one character), the browser needs to redownload the whole video.
Why different video URLs are being targeted each time? This is because the URLs that the videos are requested from, like those:
have a response header
which means, browser is not allowed at all to cache this page, every time there's a request to the server needed, the server responds with new 200 OK reponse and different params to be used to query for the video.
Other video sites may be not that restrictive and hence you can have the video cached between the loads.
I've noticed even more interesting thing when opening a video (it was ~5 MB) in IE8. In Firefox, the whole video is loaded as a one stream. In IE8, it's sent as three ~1.7 MB chunks. Perhaps some internal IE thing that it can't handle big files nicely.
How to enable caching?
One could write an addon to Firefox, or a Fiddler script, which will strip or replace appropriate cache-related headers from YouTube HTTP responses, to cheat the browser about what it's allowed to do. Then the browser will cache more aggresively and keep the video across loads, given all the other requirements are satisfied.