why do programs on linux kernel use so much more vmem than resident memory?

Question

Unlike on other operating systems of UNIX flavours, on linux, processes typically use enormous vmem compared to resident memory.

For example on my laptop:

plugin-container uses 20 times more vsz than rss, lxterminal use 40 times more vss than rss

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
petanb    2036  1.7 39.8 2027260 743108 ?      Sl   Jul01 260:16 /usr/lib/firefox/firefox
petanb    2170  3.4  1.8 668460 33716 ?        Sl   Jul01 520:11 /usr/lib/firefox/plugin-container /usr/lib/flashplugin-installer/libflashplayer.so -gr
petanb    2391  0.0  0.7 698888 14080 ?        Dl   Jul01   7:02 /usr/bin/lxterminal
petanb    4633  0.0  0.4 430568  8816 ?        Sl   Jul01   0:08 /usr/lib/notify-osd/notify-osd

look at the difference between VSZ and RSS. Why is that? I know virtual memory is bigger than resident, but why so much?

Answer 1

First of all, programs/processes do not "eat" memory. They use or are allocated memory.

VSZ (virtual memory size in kilobytes) is the sum of all virtual memory required by the process. This size is primarily an attribute of the process/program, rather than under control of the OS.

RSS is the resident set size, which is the amount of physical memory allocated to the process. This is the physical memory used to hold virtual memory pages that are resident (rather than located in the backing store, i.e. swapped out) while the process is executing. RSS will be less than or equal to the process's virtual memory size.
Note that sometimes RSS is reported as the number of pages (which typically has a size of 4096 bytes) rather than in kilobytes. So in those situations, comparing VSZ to RSS is inappropriate.

Note that both of these sizes for the process could include shared libraries as well as memory for code/text, data, heap and stack. It is up to the OS to schedule processes, and determine which pages of the process stay resident in (physical) memory and which get swapped out. The shared libraries are more likely to be kept resident than pages used exclusively by the process. A process in the sleeping state (e.g. STAT == D1 or STAT == S1) could have most of its pages swapped out and have a small RSS. It's all under the control of the OS and the dynamics of process execution.

Also note that Linux is unlike most other *nixes in that Linux (unless configured otherwise) will overcommit requests for (virtual) memory. So even though no page frames or swap space are allocated for the process (until the process tries to access this "allocated" memory), its VSS can increase.

Addendum: response to comment

is why a difference is so huge? sometimes VSZ is 20 times bigger than RSS, even on system with no swap.

You are citing a particular example without providing any details.
The overcommit feature of Linux could be a factor for a large discrepancy between VSZ and RSS if there is no swap. Does that program do malloc()s of large but unused buffers?
Maybe you will have to evaluate the memory usage of that process yourself using memory tools such as pmap and top, which will provide a bit more detail.

See "Runtime_Memory_Measurement" for some info. top documentation declares that

SWAP -- Swapped size (kb)
            The swapped out portion of a task's total virtual memory image. 

RES -- Resident size (kb)
            The non-swapped physical memory a task has used.
            RES = CODE + DATA. 

VIRT -- Virtual Image (kb)
            The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out.
            VIRT = SWAP + RES 

I don't know if these top definitions are supposed to account for virtual memory that has been committed but not yet allocated physical resources. I think it does, but the VIRT equation was simplified by omitting the (unusual) situation of having commited-but-not-yet-allocated virtual memory.

Addendum 2: response to comment

Maybe try running top or htop on your own computer, you will see that there is a number of processes that use significant amount of VSZ and almost no RSS

Did not see anything interesting on my Ubuntu system:

  PID  PR  NI  VIRT  RES  SHR S %CPU %MEM SWAP CODE DATA nDRT COMMAND                                                                                        
26379  20   0 20812 7996 3496 S    4  0.4  12m    4 2828    0 gs                                                                                             
 1082  20   0 85292  46m  15m S    2  2.5  36m 1660  30m    0 Xorg                                                                                           
 2036  20   0 22948 8756 7148 S    1  0.4  13m   40  788    0 multiload-apple                                                                                
 2411  20   0 47984  19m  10m S    1  1.0  27m 1924 7584    0 python                                                                                         
   62  20   0     0    0    0 S    0  0.0    0    0    0    0 kondemand/0                                                                                    
  930  20   0  3236 1528  792 S    0  0.1 1708  284  820    0 dbus-daemon                                                                                    
 1618  20   0  6964 3084 2244 S    0  0.2 3880  420  836    0 cupsd                                                                                          
 1971  20   0 15708 3100 2484 S    0  0.2  12m  216  10m    0 udisks-daemon                                                                                  
 2037  20   0 39316  11m 9676 S    0  0.6  26m   76 1404    0 sensors-applet                                                                                 
25733  20   0  2568 1280  956 R    0  0.1 1288   64  480    0 top                                                                                            
 2473  20   0  6712 4060 1564 S    0  0.2 2652  780 2492    0 bash                                                                        

For all of the user processes, VIRT == SWAP + RES was true, but RES == CODE + DATA was not.

I do have a few SBCs running embedded Linux, but do not have top cross-compiled for them. Looking at /proc/<pid>/stat for a shell process, there's VSZ = 580kB and RSS = 200kB (actually 50 pages), but of course /proc/meminfo reports zero bytes for swap. Maybe getting top running on this SBC might be interesting (since I don't know where top gets its numbers for swap).