2

Let's say I have 50 GiB of files that weights around 500 KiB each.

My guess is that having, for example, 5 large files of 10 GiB each with the same content archived in them would be better for hard drive performance. Am I correct?

Will there be a noticeable gain on an NTFS filesystem?

=====================================================================

Finally, which tool could I use to group the files together while retaining the ability to modify the content of the archive with zero or minor performance loss? For example, I like TrueCrypt archiving because after mounting an archive file, it creates a drive which I can use seamlessly as if it was a normal drive. The only thing with TrueCrypt is that I don't need encryption/compression, only archiving.

4
  • ...then with TrueCrypt, just set the password to "password" (and indicate that this is what it is in the filename). Jun 29, 2011 at 15:56
  • But even if the password is easy and known, it's still gonna encrypt the whole thing, which will be worse for performance than simply leaving the files unarchived.
    – user54676
    Jun 29, 2011 at 16:29
  • That's a good point. It sure would be nice if TrueCrypt had a "no encryption" option ("for testing purposese," of course). =) Jun 29, 2011 at 16:48
  • 1
    Doing this gives you the secondary problem that if your container gets corrupted/damaged at all, there's a very significant chance that you'll lose all the contained data rather than a file or two.
    – afrazier
    Jun 29, 2011 at 20:06

2 Answers 2

1

In Windows 7, you can mount a .VHD as a drive. This is the virtual hard drive format used by virtual machines and by Windows Backup (for Complete PC backup only on Windows client, and for all backups on Windows Server). No compression or encryption. Performance slowdown during ordinary use is minimal. After all, people are running whole virtual machines this way.

NTFS metadata and disk seeking can lead to substantial overhead on small files. For example, copying 10,000 files of under 10 KB each to a USB hard drive will proceed at about 300 files per second. That's 30 seconds to copy the files individually, vs. 10 seconds to copy them in a block. (The difference becomes even more striking with internal or eSATA drives, since the block throughput rate is higher. SSDs are so great at random access that it might not matter either way.)

But 500 KB files are large enough that the impact might be limited. You'd have to benchmark it and see.

1
  • That's what I was looking for. And on top of that, it's included in Windows 7, as you said, so I won't need to install or buy anything. Thanks!
    – user54676
    Jun 30, 2011 at 0:05
3

Combining files

I would expect that a single large file is only better for performance if you usually read all the data, read it sequentially and if the large file is relatively unfragmented.

TrueCrypt

Using any kind of compression or encryption will be much worse for hard drive performance.

Update:

According to an answer to this question "there will be some drop in performance, albeit a slight one." The answer refers to a Tom's Hardware article which says

The benchmark shows varying performance and highly depends on the processor, followed by the drive you are about to encrypt: AES and Twofish provide highest throughput on our Core 2 Duo notebook Dell Latitude D610. Once you start combining multiple encryption algorithms, e.g. Twofish and Serpent, performance drops considerably. While this isn’t noticeable while working with Windows and popular applications, increasing system load—such as may occur during heavy multi-tasking or when taking on intensive workloads such as video transcoding—will reduce system performance considerably.

The Wikipedia article says

When using popular desktop applications in a "reasonable manner", and with only a single encryption algorithm, the performance impact of TrueCrypt on desktop applications is not generally noticeable, though that does depend on the application, and power users may complain. Using a fast multi core processor and a fast system drive, preferably a Flash SSD, makes TrueCrypt almost transparent

I don't know of any evidence that shows Truecrypt is going to significantly be "better for hard drive performance".

2
  • Compression in and of itself isn't necessarily going to hurt anything. As long as you don't have any "solid" archiving options turned on, random access to an individual file won't be all that painful.
    – afrazier
    Jun 29, 2011 at 20:05
  • @afrazier: Thanks for pointing that out, I'll update my answer. Jun 29, 2011 at 20:10

You must log in to answer this question.