Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'd like to make the ls -laR /media/myfs on Linux as fast as possible. I'll have 1 million files on the filesystem, 2TB of total file size, and some directories containing as much as 10000 files. Which filesystem should I use and how should I configure it?

As far as I understand, the reason why ls -laR is slow because it has to stat(2) each inode (i.e. 1 million stat(2)s), and since inodes are distributed randomly on the disk, each stat(2) needs one disk seek.

Here are some solutions I had in mind, none of which I am satisfied with:

  • Create the filesystem on an SSD, because the seek operations on SSDs are fast. This wouldn't work, because a 2TB SSD doesn't exist, or it's prohibitively expensive.

  • Create a filesystem which spans on two block devices: an SSD and a disk; the disk contains file data, and the SSD contains all the metadata (including directory entries, inodes and POSIX extended attributes). Is there a filesystem which supports this? Would it survive a system crash (power outage)?

  • Use find /media/myfs on ext2, ext3 or ext4, instead of ls -laR /media/myfs, because the former can the advantage of the d_type field (see in the getdents(2) man page), so it doesn't have to stat. Unfortunately, this doesn't meet my requirements, because I need all file sizes as well, which find /media/myfs doesn't print.

  • Use a filesystem, such as VFAT, which stores inodes in the directory entries. I'd love this one, but VFAT is not reliable and flexible enough for me, and I don't know of any other filesystem which does that. Do you? Of course, storing inodes in the directory entries wouldn't work for files with a link count more than 1, but that's not a problem since I have only a few dozen such files in my use case.

  • Adjust some settings in /proc or sysctl so that inodes are locked to system memory forever. This would not speed up the first ls -laR /media/myfs, but it would make all subsequent invocations amazingly fast. How can I do this? I don't like this idea, because it doesn't speed up the first invocation, which currently takes 30 minutes. Also I'd like to lock the POSIX extended attributes in memory as well. What do I have to do for that?

  • Use a filesystem which has an online defragmentation tool, which can be instructed to relocate inodes to the the beginning of the block device. Once the relocation is done, I can run dd if=/dev/sdb of=/dev/null bs=1M count=256 to get the beginning of the block device fetched to the kernel in-memory cache without seeking, and then the stat(2) operations would be fast, because they read from the cache. Is there a way to lock those inodes and/or blocks into memory once they have been read? Which filesystem has such a defragmentation tool?

share|improve this question
    
You say "on Linux as fast as possible" Do you really mean that? What is your budget for hardware because it can be very fast if you have a legitimate reason for needing things to be so fast. –  deltaray Jan 9 '11 at 18:20
    
If your version of find has -printf (and on Linux it should), you can output the same information as ls -l (in addition to the fact that find has -ls). However, it's still going to have to do stat to get that information. Have you considered using locate or a similar scheme? –  Dennis Williamson Jan 9 '11 at 20:29
    
@deltaray: My budget is the price of an 32GB SSD drive + my time to set things up. I don't have time to develop custom software. –  pts Jan 9 '11 at 23:22
    
@Dennis Williamson: I know about locate, and it can be a useful workaround. But I'm still interested in getting an answer to my question. –  pts Jan 9 '11 at 23:24

4 Answers 4

I would just use ext4 and make sure that you have dir_index set. You can check for that flag by running this:

dumpe2fs /dev/drivepartition | grep "Filesystem features:"

The biggest problem you'll run into is just the number of files overall on the filesystem. Any operation you run across the filesystem will have to look at each file. This is the case with any filesystem. 10,000 files in a directory may seem like a lot, but I find that filesystems don't get slow until you get to 40,000 files or more and that's really an older symptom of filesystems like ext2.

It sounds like you're trying to do something specific rather than just have a general purpose filesystem. If you can explain what you're trying to do, we can probably suggest a way to optimize your data. For instance, a database.

share|improve this answer
    
Thank you for composing your answer. I know about the dir_index flag, and it doesn't help me (i.e. it's still too slow with the flag enabled), because it doesn't prevent the stat(2) calls and the disk seeks. –  pts Jan 10 '11 at 9:36
    
My use case is interactive: I go to directories in Midnight Commander, then I view, copy and move some some files. If the directory contains 10000 files, sometimes it takes 50 seconds to refresh it in Midnight Commander (because of the stat(2) calls). –  pts Jan 10 '11 at 9:38
    
Again I ask, why do you have 10,000 files in a directory? –  deltaray Jan 10 '11 at 15:38
    
Some of my scripts and programs create files into the same directory. Every week I open those directories manually, and move the files away, possibly to smaller directories. However, in this question, I am not interested in alternative designs. I am only interested in what I asked. –  pts Jan 10 '11 at 18:56
    
If your budget is limited then you better start looking into alternative designs. You can try XFS or JFS and see if you see an improvement, but I think that if your budget is limited, you should start thinking of a better design for your data. –  deltaray Jan 10 '11 at 22:06

I'll trade you my answer to your question for your answer to mine: What knobs have to be fiddled in /proc or /sys to keep all the inodes in memory?

Now for my answer to your question:

I'm struggling with a similar-ish issue, where I'm trying to get ls -l to work quickly over NFS for a directory with a few thousand files when the server is heavily loaded.

A NetApp performs the task brilliantly; everything else I've tried so far doesn't.

Researching this, I've found a few filesystems that separate metadata from data, but they all have some shortcomings:

  • dualfs: Has some patches available for 2.4.19 but not much else.
  • lustre: ls -l is a worst-case scenario because all the metadata except the file size is stored on the metadata server.
  • QFS for Solaris, StorNext/Xsan: Not known for great metadata performance without a substantial investment.

So that won't help (unless you can revive dualfs).

The best answer in your case is to increase your spindle count as much as possible. The ugliest - but cheapest and most practical - way to do this is to get an enterprise-class JBOD (or two) and fiber channel card off of Ebay that are a few years old. If you look hard, you should be able to keep your costs under $500 or so. The search terms "146gb" and "73gb" will be of great help. You should be able to convince a seller to make a deal on something like this, since they've got a bunch of them sitting around and hardly any interested buyers:

http://cgi.ebay.ca/StorageTek-Fibre-Channel-2TB-14-Bay-HDD-Array-JBOD-NAS-/120654381562?pt=UK_Computing_Networking_SM&hash=item1c178fc1fa#ht_2805wt_1056

Set up a RAID-0 stripe across all the drives. Back up your data religiously, because one or two of the drives will inevitably fail. Use tar for the backup instead of cp or rsync so that the receiving single drive won't have to deal with the millions of inodes.

This is the single cheapest way I've found (at this particular historical moment, anyway) to increase IOPs for filesystems in the 2-4TB range.

Hope that helps - or is at least interesting!

share|improve this answer
    
About keeping inodes on memory on Linux: try sudo sysctl -w vm.vfs_cache_pressure=1 (or =0). I got this hint from a forum, I've never tried. –  pts Apr 8 '11 at 19:39

the disk contains file data, and the SSD contains all the metadata ... Is there a filesystem which supports this?

btrfs supports this to some extent, btrfs Wiki. One can specify raid1 for the metadata (and raid0 for data - most data will end up on the large HDD) so that the SSD will always have a copy of the metadata for reading (I have no idea how clever btrfs will be in selecting the source for reading metadata). I have not seen any benchmarks for such a setup.

share|improve this answer
    
Thank you for mentioning btrfs. Unfortunately I couldn't find the feature you mentioned. spinics.net/lists/linux-btrfs/msg05617.html says that in July 2010 this was not possible. Could you please cite your source stating otherwise. –  pts Jan 24 '13 at 7:37

No answer, unfortunately, although I did google for an answer for the last half an hour.

Create a filesystem which spans on two block devices: an SSD and a disk; the disk contains file data, and the SSD contains all the metadata (including directory entries, inodes and POSIX extended attributes). Is there a filesystem which supports this? Would it survive a system crash (power outage)?

Exactly what I also would like.

For the links, see this pastebin, because I'm not allowed to post more than one link...

http://www.notehub.org/2014/10/2/external-metadata-more-information

Multi-device support from btrfs is discussed here:

Btrfs: Working with multiple devices, by By Jonathan Corbet, December 30, 2013 (LWN), [link][1]

But although you can mirror the metadata (-m raid1) to a SSD, you are then forced to also use the SSD for data (-d raid0) storage, at least partially.

The good news is that there is work being done:

Dedicated metadata drives Jan Schmidt and Arne Jansen (Not in kernel yet) We're able to split data and metadata IO very easily. Metadata tends to be dominated by seeks and for many applications it makes sense to put the metadata onto faster SSDs. [link][2]

If you are willing to use IBM's proprietary General Parallel File System (GPFS), then this is already possible, it seems. Read "How to migrate all GPFS filesystem metadata to SSDs": [link][3]

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.