When should I use
/dev/shm/ and when should I use
/tmp/? Can I always rely on them both being there on UNIX?
/dev/shm is a temporary file storage filesystem, i.e., tmpfs, that uses RAM for the backing store.
It can function as a shared memory implementation that facilitates IPC.
Recent 2.6 Linux kernel builds have started to offer /dev/shm as shared memory in the form of a ramdisk, more specifically as a world-writable directory that is stored in memory with a defined limit in /etc/default/tmpfs. /dev/shm support is completely optional within the kernel config file. It is included by default in both Fedora and Ubuntu distributions, where it is most extensively used by the Pulseaudio application. (Emphasis added.)
/tmp is the location for temporary files as defined in the Filesystem Hierarchy Standard, which is followed by almost all Unix and Linux distributions.
Since RAM is significantly faster than disk storage, you can use
/dev/shm instead of
/tmp for the performance boost, if your process is I/O intensive and extensively uses temporary files.
To answer your questions: No, you cannot always rely on
/dev/shm being present, certainly not on machines strapped for memory. You should use
/tmp unless you have a very good reason for using
/tmp can be part of the
/ filesystem instead of a separate mount, and hence can grow as required. The size of
/dev/shm is limited by excess RAM on the system, and hence you're more likely to run out of space on this filesystem.
1I will be using it to redirect output from a commands' standard error output to a file. Then I will read this file and process it. I will be doing this several thousand times (it's part of the condition of a loop construct). I thought memory would be nice in this case. But I also want it to be portable. I guess I'll check if
/dev/shmexists, use it if it does, or fallback to
/tmp. Does that sound good? Sep 23, 2009 at 16:04
1I'd also add a check for the minimum size and current usage level of /dev/shm to guard against inadvertently filling it up.– user4358Sep 23, 2009 at 21:01
6Under Linux 2.6 and later the /dev/shm is required to be mounted for the POSIX shared memory system calls like shm_open() to work. In other words some programs will break if its not mounted - so it should be. It is not just a RAM disk. So you should make sure some of /dev/shm is free.– EdHNov 30, 2012 at 21:07
13There is no performance boost by using
/dev/shmis memory (tmpfs) backed by the disk (swap).
/var/tmpis memory (disk cache) backed by the disk (on-disk filesystem). In practice, performance is about the same (tmpfs has a slight edge but not enough to matter).
/tmpmay be tmpfs or not depending on how the administrator configured it. There is no good reason to use
/dev/shmin your scripts. Jun 18, 2013 at 7:13
4@GaretClaborn There's plenty of good reasons to use memory backed by swap, but that's called normal process memory. If you're using a file, it's called a filesystem, and all filesystems are memory (cache), which is backed by swap if the filesystem is something like tmpfs. Allocating disk space between swap and other storage areas is typically in the real of the administrator. If an application wants files that tend to remain in RAM,
/tmpis the normal location (with
$TMPDIRto override). The choice to make
/tmpbacked by swap, other disk space or nothing is the administrator's. Jun 5, 2014 at 14:09
In descending order of
┌───────────┬──────────────┬────────────────┐ │ /dev/shm │ always tmpfs │ Linux specific │ ├───────────┼──────────────┼────────────────┤ │ /tmp │ can be tmpfs │ FHS 1.0 │ ├───────────┼──────────────┼────────────────┤ │ /var/tmp │ never tmpfs │ FHS 1.0 │ └───────────┴──────────────┴────────────────┘
Since you are asking about a Linux specific tmpfs mountpoint versus a portably defined directory that may be tmpfs (depending on your sysadmin and what's default for your distro), your question has two aspects, which other answers have emphasized differently:
- Appropriate use of various tmp directories
- Appropriate use of tmpfs
Appropriate use of various tmp directories
Based on the ancient Filesystem Hierarchy Standard and what Systemd says about the matter.
- When in doubt, use
/var/tmpfor data that should persist across reboots.
/var/tmpfor large data that may not easily fit in RAM (assuming that
/var/tmphas more available space – usually a fair assumption).
/dev/shmonly as a side-effect of calling
shm_open(). The intended audience is bounded buffers that are endlessly overwritten. So this is for long lived files whose content is volatile and not terribly large.
- Definitely don't use
/dev/shmfor executables (of any kind), as it's commonly mounted
- If still in doubt, provide a way for the user to override. For the least amount of surprise, do like
mktempand honor the
Where tmpfs excels
tmpfs performance is deceptive. You will find workloads that are faster on tmpfs, and this is not because RAM is faster than disk: All filesystems are cached in RAM – the page cache! Rather, it is a sign that the workload is doing something that defeats the page cache. And of the worse things a process can do in this regard is syncing to disk way more often than necessary.
fsync is a no-op on tmpfs. This syscall tells the OS to flush its page cache for a file, all the way down to flushing the write cache of the relevant storage device, all while blocking the program that issued it from making any progress at all – a very crude write barrier. It is a necessary tool in the box only because storage protocols aren't made with transactions in mind. And the caching is there in the first place to make it possible for programs to perform millions of small writes to a file without noticing how slow it actually is to write to a storage device – all actual writing happens asynchronously, or until
fsync is called, which is the only place where write performance is directly felt by the program.
So if you find yourself using tmpfs (or eatmydata) just to defeat fsync, then you (or some other developer in the chain) are doing something wrong. It means that the transactions toward the storage device are unnecessarily fine grained for your purpose – you are clearly willing to skip some savepoints for performance, as you have now gone to the extreme of sabotaging them all – seldom the best compromise. Also, it is here in transaction performance land where some of the greatest benefits of having an SSD are – any SSD worth its salt is going to perform out-of-this-world compared to what a spinning disk can possibly take (7200 rpm = 120 Hz, if notihing else is accessing it). Flash memory cards also vary widely on this metric (it is a tradeoff with sequential performance, and the SD card class rating only considers the latter). So beware, developers with blazing fast SSDs, not to force your users into this use case!
Wanna hear a ridiculous story? My first
fsync lesson: I had a job that involved routinely "upgrading" a bunch of Sqlite databases (kept as testcases) to an ever-changing current format. The "upgrade" framework would run a bunch of scripts, making at least one transaction each, to upgrade one database. Of course, I upgraded my databases in parallel (8 in parallel, since I was blessed with a mighty 8 core CPU). But as I found out, there was no parallelization speedup whatsoever (rather a slight hit) because the process was entirely IO bound. Hilariously, wrapping the upgrade framework in a script that copied each database to
/dev/shm, upgraded it there, and copied it back to disk was like 100 times faster (still with 8 in parallel). As a bonus, the PC was usable too, while upgrading databases.
Where tmpfs is appropriate
The appropriate use of tmpfs is to avoid unnecessary writing of volatile data. Effectively disabling writeback, like setting
/proc/sys/vm/dirty_writeback_centisecs to infinity on a regular filesystem.
This has very little to do with performance, and failing this is a much smaller concern than abusing fsync: The writeback timeout determines how lazily the disk content is updated after the pagecache content, and the default of 5 seconds is a long time for a computer – an application can overwrite a file as frequently as it wants, in pagecache, but the content on disk is only updated about once every 5 seconds. Unless the application forces it through with fsync, that is. Think about how many times an application can output a small file in this time, and you see why fsyncing every single one would be a much bigger problem.
What tmpfs can not help you with
- Read performance. If your data is hot (which it better be if you consider keeping it in tmpfs), you will hit the pagecache anyway. The difference is when not hitting the pagecache; if this is the case, go to "Where tmpfs sux", below.
- Short lived files. These can live their entire lives in the pagecache (as dirty pages) before ever being written out. Unless you force it with
Where tmpfs sux
Keeping cold data. You might be tempted to think that serving files out of swap is just as efficient as a normal filesystem, but there are a couple of reasons why it isn't:
- The simplest reason: There is nothing that contemporary storage devices (be it harddisk or flash based) loves more than reading fairly sequential files neatly organized by a proper filesystem. Swapping in 4KiB blocks is unlikely to improve on that.
- The hidden cost: Swapping out. Tmpfs pages are dirty — they need to be written somewhere (to swap) to be evicted from pagecache, as opposed to file backed clean pages that can be dropped instantly. This is an extra write penalty on everything else that competes for memory – affects something else at a different time than the use of those tmpfs pages.
In my ubuntu 14.04 /dev/shm is link to /run/shm, that has filesystem "none" according to command df. Size is about 2G, though.– jarnoDec 14, 2016 at 8:41
4@jarno Firstly, economizing the number of tmpfs mountpoints, I'd call an implementation detail. Secondly, don't let the device name confuse you – look in /proc/mounts (that's the right place to look), and you will see that the type is "tmpfs" while the device is what's "none" here. Yes, the device name means nothing in tmpfs – you can
mount -t tmpfs "jarno is great" /mnt/jarnoif you like! Thirdly, the default size is half the amount of RAM – I bet you have 4GiB RAM. Dec 22, 2016 at 19:04
1Is there an option that allocates a fixed RAM size and promises to never use swap?– palswimJul 28, 2017 at 21:44
@palswim: That would be a ramdisk. I don't see an option for that in tmpfs, other than that tmpfs' predecessor didn't support swapping. Processes can lock their pages in ram, which is arguably less crazy than locking tmpfs pages in ram, considering that the OOM killer is unable to free up the latter, should you run out of memory. Aug 17, 2017 at 14:18
1The sqlite upgrade scripts may have had autocommit on? Or commit'ed too often? I've seen a huge difference in sqlite performance with and without autocommit. Running on memory (/dev/shm) alleviates committing too often. Jun 18, 2021 at 9:47
Okay, here's the reality.
Both tmpfs and a normal filesystem are a memory cache over disk.
The tmpfs uses memory and swapspace as it's backing store a filesystem uses a specific area of disk, neither is limited in the size the filesystem can be, it is quite possible to have a 200GB tmpfs on a machine with less than a GB of ram if you have enough swapspace.
The difference is in when data is written to the disk. For a tmpfs the data is written ONLY when memory gets too full or the data unlikely to be used soon. OTOH most normal Linux filesystems are designed to always have a more or less consistent set of data on the disk so if the user pulls the plug they don't lose everything.
Personally, I'm used to having operating systems that don't crash and UPS systems (eg: laptop batteries) so I think the ext2/3 filesystems are too paranoid with their 5-10 second checkpoint interval. The ext4 filesystem is better with a 10 minute checkpoint, except it treats user data as second class and doesn't protect it. (ext3 is the same but you don't notice it because of the 5 second checkpoint)
This frequent checkpointing means that unnecessary data is being continually written to disk, even for /tmp.
So the result is you need to create swap space as big as you need your /tmp to be (even if you have to create a swapfile) and use that space to mount a tmpfs of the required size onto /tmp.
NEVER use /dev/shm.
Unless, you're using it for very small (probably mmap'd) IPC files and you are sure that it exists (it's not a standard) and the machine has more than enough memory + swap available.
38Agreed, except for the conclusion, "NEVER use /dev/shm". You want to use /dev/shm in cases where you don't want a file to be written to the disk at all, and you want to minimize disk i/o. For example, I need to download very large zip files from an FTP server, unzip them, and then import them into a database. I unzip to /dev/shm so that for both the unzip and the import operations the HDD only needs to perform half the operation, rather than moving back and forth between source and destination. It speeds up the process immensely. That's one example of many, but I agree that it's a niche tool. Dec 11, 2014 at 23:13
See 0pointer.net/blog/projects/tmp.html. You can use
$XDG_RUNTIME_DIRfor temporarily storing downloaded files. I know this answer is many years old, so is the link I posted. Apr 22, 2020 at 8:24
Use /tmp/ for temporary files. Use /dev/shm/ when you want shared memory (ie, interprocess communication through files).
You can rely on /tmp/ being there, but /dev/shm/ is a relatively recent Linux only thing.
Isn't there a performance aspect as well? As /dev/shm is most often mounted as a tmpfs volume and essentially a RAM-disk? Sep 23, 2009 at 9:37
You can also mount /tmp as a tmpfs filesystem, I do so on my netbook to speed some things up by reducing writes to the (slow) SSD. There are disadvantages to doing so, of course (mainly the RAM use, but my netbook has far more RAM than it generally needs anyway). Sep 23, 2009 at 10:55
For my specific case I would use it for a sort of process communication. I capture the output of standard error from an application and act on the contents (and I still need the standard output untouched, so I can't do any
1>/dev/null 2>&1. I would do this several thousand times so a tmpfs would be nice. However if I release the script I can't rely on tmpfs being used for
/tmpas I think it's not that common. If it's more common for
/dev/shmthen that's better for me. But I'm looking for guidelines regarding portability etc. Sep 23, 2009 at 16:00
Another time when you should use /dev/shm (for Linux 2.6 and above) is when you need a guaranteed tmpfs file system because you don't know if you can write to disk.
A monitoring system I'm familiar with needs to write out temporary files while building its report for submission to a central server. It's far more likely in practice that something will prevent writes to a file system (either out of disk space or an underlying RAID failure has pushed the system into a hardware read-only mode) but you'll still be able to limp along to alert about it than if something spirals all available memory such that tmpfs will be unusable (and the box won't be dead). In cases like this, a monitoring system will prefer writing out to RAM so as to potentially be able to send an alert about a full disk or dead/dying hardware.
/dev/shm is used for shared virtual memory system specific device drivers and programs.
If you are creating a program that requires a virtual memory heap that should be mapped to virtual memory. This goes double so if you need multiple processes or threads to be able to safely access that memory.
The fact is that just because the driver uses a special version of tmpfs for it, doesn't mean you should use it as a generic tmpfs partition. Instead, you should just create another tmpfs partition if you want one for your temporary directory.
In PERL, having 8GB minimum on any machine (all running Linux Mint), I am of what I think is a good habit of doing DB_File-based (data structure in a file) complex algorithms with millions of reads and writes using /dev/shm
In other languages, not having gigether everywhere, to avoid the starts and stops in network transfer (working locally on a file that is located on a server in a client-server atmosphere), using a batch file of some type, I will copy the whole (300-900MB) file at once to /dev/shm, run the program with output to /dev/shm, write the results back to the server, and delete from /dev/shm
Naturally, if I had less RAM, I would not be doing this. Ordinarily, the in-memory file system of /dev/shm reads as a size being one half of your available RAM. However, ordinary use of RAM is constant. So you really couldn't do this on a device with 2GB or less. To turn paraphrase to hyperbole, there is often things in RAM that even the system doesn't report well.
(I think this is in the spirit of what was originally asked.) What I mean basically is that I'm comfortable using /dev/shm as a RAM disk as long as I have sufficient memory. If it is inefficient to do so somehow, that should not dissuade you from doing so, but should trigger a question like "How can I have a ram disk on linux?". The answer is /dev/shm Sep 27, 2015 at 21:44