Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

For instance, I have video file which is 11.8 Gb, but my RAM memory only 2 Gb. How does VLC (or other software) handle it? How do they load it into memory? I used VMMap tool (from sysinternals) to take a look at memory, and I saw:

Private 160000K

Working set 100000K

Obviously, it's much less than 11.8 Gb -So how did it happen?

This question is not only about video. I'd like to know how Computer, in general, handles very large files.

share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

This is so complicated that it's extremely difficult to even begin explaining, so I'll just mention few basic ways programs can work.

First and the most obvious way which is also often the slowest is to work directly with the file on disk. Basically each block on disk has its own logical address and an application can directly work with data on the disk. So if I'm designing a simple text editor, I could load a screen of text into video memory from the disk and write any changes directly to disk as soon as they're made. This type of approach is (as far as I know) almost never used today because of its numerous downsides. First problem with it is that disk are so very slow when compared to RAM that the CPU would practically spend all of its time waiting for the disk to finish keeping up with data. The up side is that we're using almost no RAM, as all data from the disk can be transferred directly to the RAM in video card. On top of all that, you have modern operating systems that make direct access to hardware even slower and in many cases impossible.

Next we have the (unfortunately) common and most obvious solution for the problem of slow disk access: We'll just copy the whole file to RAM and work the the RAM copy. When we're finished, we'll somehow synchronize the RAM version with the version on disk and solve the problem. Modern operating systems make this relatively easy as the application programmer can use services provided by the OS to update the file without thinking too much about how it's really done. The major upside of this approach is speed. RAM is (compared to disks) extremely fast and disks generally work better when larger amount of data needs to be transferred. Furthermore this approach leaves disk available for other applications to use and you can edit the file while another application is working with the disk. The downside is that it is assumed that the whole file can be loaded into RAM in reasonable amount of time and that the file will leave enough space for other tasks in RAM. Sometimes this is not true. For example, I once had to open a ~3.5 GiB text file and it turned out that most applications assume that the text file will fit in RAM.

Next approach which in general is used when we are working with application that expect large files is to load a part of the file into RAM and work with it. When we're finished, we'll save that part to the disk and read the next part. How exactly this works depends on the structure of the file itself.
In some file types, you may find an index at the start of the file which you can load into RAM and use it to determine later the logical addresses of the interesting parts of the file. In some other file types, you may need to search the whole file for the section which has the data you need and then just load that part of the file into RAM.

This approach also provides space for clever optimizations like allowing editing of a part of the file while another part is being loaded into RAM in background in order to minimize the wait time required to open the file and so on.

So in the video file example, some data about the format itself would be encoded at the start of the fine and later on, the program which plays the file will only need have in memory the part of the file which is currently being played. In order to make the playback more smooth, programs will also keep part of the file which is yet to be played into RAM. Usually it's not easy to exactly determine how long it's going to take for disk to access the data. For example due to fragmentation, a part of the file may be at the beginning of the disk while a part may be at the end of the disk. Also at the same time while the video is being played, another application may try to write large amounts of data onto the disk. Since the video player already has some buffer in RAM, the playback should continue without visible interruptions.

This approach has the upside of using less RAM than previous and at the same time being reasonably fast for uses which the programmer predicts. The downside is that you're relying on the programmer to predict which parts of the file will be commonly used and how and sometimes the expected usage pattern can be different than real usage pattern. The other downside is that it takes effort to precisely determine which part of the file needs to be in RAM and how large that part needs to be. If the part is too small, you're not gaining enough speed and if the part is too large, you're taking hogging up RAM.

So to summarize the 3 options I described: The first one would be kid in elementary school who's underlining each letter he sees with a pencil while struggling to read a word.

The second would be printing the whole text on a single page and if the page is as big as a wall, then we may run into some problems.

The third option would be like reading from a book. You open the book at a certain page and right next to it you have another page open! When you finish reading both, you move on to the next pair.

Do note that in this answer I haven't discussed much about countless caches and abstraction layers that exist in modern computers between the disk, RAM and processor. For example in a real world situation, if you have one program which is doing heavy disk access and another that tries to save a small file, that file may be stored somewhere in RAM in a cache until the disk has enough free time to write it. Also the disk itself has its won internal cache and it may store the file there for some time before writing it to disk. Also when reading, the OS itself may load into RAM more disk blocks than an application asked for because it (correctly or not) predicted that application may need them soon. Same thing goes for the disk cache too. Then it may turn out that the disk isn't actually a disk but a RAID and that we have a cache on the raid controller and on each individual disk and so on.

share|improve this answer
add comment

When you read a book, you don't remember each and every word. You remember the improtant details that are relevant to the rest of the story. If you need to remember more detail, you can flip back and read a page again. You find the page by looking up the page number in the index or contents.

When a computer plays a video, it reads so many seconds of video data so that it has enough data to begin processing. It starts to generate video frames and queues them to be displayed on the monitor. It forgets the video data after processing it as it is no longer required. It loads more video data in it's place. After a video frame is displayed, it too is forgotten as it is no longer required. If the user rewinds the video, the computer rewinds to the appropriate point in the file and repeats the process of reading video data and generating and queueing video frames.

For other file formats, the computer will act in a similar manner. It will only read and process the part of the file that is currently requested.

share|improve this answer
1  
It will only read and process the part of the file that is currently requested. That's unfortunately, not always the case. Some programs may assume that entire file will fit into RAM and try to load it whole. Fun times happen if it can't. –  AndrejaKo Nov 25 '11 at 1:07
1  
Sure, but that's probably the wrong program for the job. When I say "a computer", I should be saying "a program designed to read large files". –  Hand-E-Food Nov 25 '11 at 2:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.