15

In our office we use a RAID5 of SSDs as a network share on a linux server. This share is accessed as a network drive from Windows-PCs and Mac. Sometimes this network share becomes awfully slow in terms of access times and transfer speed.

I am not the admin and therefore do not have full insight into the system.

One of the admins now proposed that this may have to do with the number of files that are stored on the network share. Some folders contain millions of few kB files.

Does the access speed depend on the number of files on a network share?

5
  • 9
    Related: superuser.com/questions/1733104/… also check the comments. Having millions of files in a directory, or even having had a directory with millions of files, can affect filesystem performance when that directory is concerned.
    – Mokubai
    Aug 14, 2022 at 12:25
  • Do you know which filesystem the network share is running?
    – qwr
    Aug 15, 2022 at 3:17
  • 2
    @Robert The advice is universal. Rebuild time on SSDs is less, but not zero. If an SSD is failing due to age, others bought at the same time are likely to as well. If failing due to a defect, if others are from the same batch they have an increased risk of similar failure. Why take the chance when other formats exist?
    – Logarr
    Aug 15, 2022 at 3:47
  • 3
    @Logarr: SSD's are not mechanical. They do wear, but in an entirely different way, and the associated failure mode from expected wear is pretty benign (becomes read-only). The typical failure mode is not wear-related, though, and therefore not really related to age. Still, a lightning strike can easily take out 2 SSD's, so the usual advice still holds: RAID is not backup.
    – MSalters
    Aug 15, 2022 at 5:46
  • Under pretty much any OS, you should be able to access those files via a console. You can try writing down the name and path of one such file one day. Come back a week later (to "hope" that any cache is cleared), and try copying that file to your local disk from console by giving its absolute path. That should bypass any directory content enumeration, and I presume this is going to be very fast. Aug 15, 2022 at 17:32

4 Answers 4

32

It's not the sheer number of files on the drive, so much as how many are in any given folder.

Every time someone accesses a folder, the contents must be read so the file list can be presented. This is also independent of the file sizes; only the titles, created/modified dates & other outwardly-visible info needs to be fetched.
Icon caches could also suffer heavy impact, if thumbnails are used.

Splitting these gigantic folders into sub-sets might be just what the structure needs.

1
  • 5
    Adding to this, the term to search for the type of split the OP probably wants is ‘directory hashing’. The usual setup is to create subdirectories with names matching prefixes of file names, and then sort the files into the corresponding folders. Many mail servers, for example, do this for exactly this performance aspect. Aug 15, 2022 at 1:39
9
  1. The speed of listing files obviously does depend on the number of files to be listed.

  2. The speed of opening a specific file (i.e. starting the retrieval) can depend on the number of files.

    Depending on which filesystem is being used on the server (e.g. NTFS, XFS, ext4, ZFS), it will use different data structures to store the list of files in each directory – some of which are noticeably better at handling massive lists than others (e.g. B-trees vs hashtables vs linear lists).

    Every time a new file is opened (or otherwise touched), the server needs to find it within that directory, and this may take some time. (Especially if the directory listing isn't cached in memory and needs to be read from an HDD.)

    With millions of files, you should definitely consider sharding them into subdirectories, e.g. based on the first few letters of the filename (similar to what you might see in .git/objects/ of a Git repository).

  3. The speed of transferring a file's contents (not including the time needed to open it) doesn't depend on the number of files in that directory at all.

    It does depend on how much the disks need to seek (if they're mechanical), which is especially bad for many tiny files.

If you're transferring thousands of tiny files, I guess most of the time will be spent in and – if the server is using HDDs – physically seeking the HDD heads back and forth from one tiny file to another, and from one metadata entry to another.

4
  • I think your point 3 might be incorrect or need an asterisk. I’ve seen countless times in my career that copying 1,000 1 MB files takes a lot longer than copying 1 1 GB file. I’ve always assumed that reading 1,000 file headers and pointers, etc. and then creating new headers and pointers, etc at the destination is the slowdown. So the raw network throughout is the same, but the net user experience is very different. Aug 15, 2022 at 13:03
  • @ToddWilcox I think what point 3 means is that the time to copy a file from one folder to another won't change no matter if the source folder has 1, 500, or 1,000,000 files; in other words it's not about how many files you're transfering but how many files do the source folder have in total.
    – Josh Part
    Aug 15, 2022 at 16:06
  • @JoshPart Ah good point, you may be right. Even in that case I think there are some unusual situations where it could have an effect. If there are a huge number of files in one folder and the copy command uses a wildcard to select the source file, even if there’s only one matching source file, it could take much longer for the file system to find the right file to copy. But as I said, not a typical situation. I only mention it because I’ve encountered such things before and struggled to explain to management what the big delay was. Aug 15, 2022 at 16:09
  • @ToddWilcox I suspect your case is a bit different. It's not necessarily about the listing itself (1000 files isn't a huge amount), but more likely related to "random access" versus (in a "defragmented" drive) sequential access. When accessing a directory and listing files, you don't need to access the list of "storage blocks" where the content is. This is an extra thing that takes time. Finally, if we think 1000 files in 1 single 1 MB fragment vs 1 file of 1 GB fragment, you'll still "seek" 1000 more times, as you end up having 1000 fragments vs 1. Then on top of it, cache may come into play. Aug 15, 2022 at 17:25
6

You didn't say whether the server was Windows or Linux, but at least in Linux based file systems, large directories are certainly slow. If you create millions of files in one directory, the directory index grows. You can actually see that if you do ls -lhd <dir>. And directories only grow; they don't get smaller.

I manage a system that deals with many queue files, and to avoid slowdowns because of that, there are two things I do:

  • Split up the millions of file over various sub directories. This is a very common practice. If you look at Postfix SMTP server for instance, you'll see the queue dir is subdivided into sub directories, based on the first letter (this can be done with hashing or any algorithm you want).
  • Occasionally recreate all the sub directories. There are events that cause even those sub dirs to grow, and once a directory is dozens or a hundreds of megabytes in size (not the contents, just the dir index), it slows down all access on it.

So, avoid millions of files in one dir and put them in subdirectories.

When you're talking about millions of files spread out over many sub directories, that shouldn't be a factor.

1
  • He did say it was a Linux server.
    – chicks
    Aug 15, 2022 at 18:09
1

A likely bottleneck is the network interface.

The answer to the question as asked is "it depends". It depends on the OS, Filesystem, file sharing protocol, RAM, SSD Interface, if Encryption at rest is used and how, the RAID controller among other things.

It is possible that the number of files on the drive are impacting performance - bit this is likely only an issue if files are only read occassionally and/or the server is very memory constrained - the file system pointers are typically kept in memory and as the disk is SSD, "seek times" are a non issue.

Its also possible one or more SSDs is nearing its end of lufe, or that its not handling TRIM correctly, in which case it could be greatly slowing reads and particulsrly writes, possibly disproportionately affecting access to other disks as data us striped across all the disks.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .