12

I noticed that copying 24Mb worth of data from one folder to another took about 30 seconds because (I'm assuming this is the reason) it was over 1,000 separate files. Copying 24Mb shouldn't take so long. Why does the number of files make a different?

I'm running Windows 7 on a MacBook (4GB ram, Intel(R) Core(TM)2 Duo CPU P7450 @ 2.13GHz, 32-bit Operating System)

EDIT: NTFS is the file system used on the drive

5

The HDD does not have an exact transfer rate, it depends on proper maintenance, ie that is not fragmented, nor has bad sectors, etc...

If HDD is SATA 2 and it's the same partition, it's only the speed of data transfer.

If there are two partitions in the same HDD, it is not required this data transfer between the bus and motherboard, ie it loads into the buffer. (then depends of hdd buffer size.)

But for every file copied, the system must keep its index in the HDD's MFT (Master File Table), which makes the copying process slower if you copy many files. And if you have any anti-virus, it will scan each file that is copied. And if you have enabled the microsoft search file indexing (or any other file indexing service), the result will be worse.

I think there must be many other reasons why the copy of many files is slower, but these should be the main.

| improve this answer | |
  • Disabling the antivirus program made the extract 10x faster. After that scanned the files for viruses. – fat_mike Nov 27 '16 at 16:05
30

Why does the number of files make a different?

Apparently you are focusing solely on the "copy the data" aspect of "copy a file". A file is more than just the data; it is an entity in a filesystem. A file has a name and attributes and permissions. All of this additional information about the file has to be duplicated along with the data when the "file is copied". There is a significant amount of disk I/O to perform this filesystem overhead.

The procedure for copying one (1) file in a generic filesystem would be something like:

  • Find the source file in the filesystem. (a)
  • Read from disk the directory entry for the source file.
  • Verify read permissions.
  • Find the destination file in the filesystem. (b)
  • Verify write permissions in the destination directory.
  • Expand the directory if necessary to accommodate the new file. (c)
  • Update the directory on disk. (c1)
  • Find free blocks, allocate them and update the table again. (d)
  • Read file data and copy to destination file (i.e. copy the "file").
  • Update the directory entry for the new file with (size and time). (e)
  • Update the access time of the source directory entry. (f)

(a) At the very least this means searching the current directory. Or the path might start at the root of the filesystem, and several levels of directories have to be traversed.

(b) At the very least this means searching the current directory. Or the path might start at the root of the filesystem, and several levels of directories have to be traversed. If the destination file already exists, then determine how the copy should proceed or abort. If the destination file does not exist, then a new directory entry must be created, and maybe this involves expanding the directory (i.e. file block (aka cluster) allocation overhead).

(c) If the directory has to be expanded, allocate a new block by finding a free block, modify the allocation table with the new allocation, and then write the block(s) out to disk. Since most filesystems maintain multiple copies of the allocation table, then that means multiple writes to disk.
(c1) Once the destination directory is located, read the directory block from disk, modify it with the new directory entry for the copied file, and then write the block out to disk.

(d) In order to copy the file, allocate blocks by finding free blocks, modify the allocation table with the new allocations, and then write the block(s) out to disk. Since most filesystems maintain multiple copies of the allocation table, then that means multiple writes to disk. In order to maintain data integrity, the filesystem may not try to coalesce (delay and merge) disk write operations for directories and allocation tables, but rather perform the write operations immediately as the new files are created and block allocated.

(e) Once the data copy is complele, update the new directory entry for the copied file with the proper file length and timestamps, and then write the directory block out to disk.

(f) Update the source directory entry with a new "access" timestamp and then write the directory block out to disk.

So instead of just one file, your question is asking if doing all this stuff for one thousand files might add to the time it takes to just copy the data portion of the files? If you copy just one file of 24MB, then you will have something to compare with your copy time of one thousand files.

When backing-up a filesystem, copying the individual files to another filesystem on a disk or partition is rarely employed because it's a rather slow process as you have discovered. A faster method is to create & write a single archive file that holds the source directory entries and file contents in a special file format; backup programs and the *nix command 'tar' can output such an archive file. (Note that 'tar' just handles archive files and does not use compression like archival+compression utilities.) The fastest method of backup is to write to a block device (rather than a filesystem on a device), so that the source filesystem is ignored (treated as more data) and a block-by-block image copy of the source device can be performed.

| improve this answer | |
-1

Read FAT, re-position heads, open file on changing file - it's all require time

| improve this answer | |
  • It's an NTFS drive – Trindaz Oct 9 '11 at 4:06
  • In NTFS, all file data—file name, creation date, access permissions, and contents—are stored as metadata in the Master File Table. The Master File Table (MFT) contains metadata about every file, directory, and metafile on an NTFS volume. It includes filenames, locations, size, and permissions. A directory entry consists of a filename and a "file ID" which is the record number representing the file in the Master File Table. The file ID also contains a reuse count to detect stale references. – Lazy Badger Oct 9 '11 at 4:16
  • So is your answer to the question based on the fact that it has to do so much extra work creating records for the file name, location, size, permissions etc? That's where all the extra work comes from. – Trindaz Oct 9 '11 at 4:33
  • Creating new and reading existing – Lazy Badger Oct 9 '11 at 5:57
-1

Here's a conceptual analogy:

(please remember dear internet, that analogies are flawed. Feel free to point out how in comments.)

Assumptions:

  • The agent of information transfer == some kid, let's call him Sam, is moving eggs from one basket into another basket.
  • The medium of information transfer == air I guess. Not really important here.
  • The eggs come from various species, including like...dinosaurs...& mythical under-water seabeasts & dragons. (the eggs can be really big)
  • File-size == egg-volume. Think robbin's egg vs. draggon's egg.

Analogy:

So Sam has to move eggs from one basket to the other because he works on a farm & whoever gives him tasks is mean. There are various egg-laying animals on this farm, including dragons & mythical sea-beasts; Sam thinks this is awesome for several reasons. (not really important but this is my analogy)
Sam is chillin by the river with his 2 basketes, one of the baskets is the one with eggs inside of it. The eggs in the basket range in size from

  • a robbin's egg taking up like 100 bytes of space in the basket

to

  • Kraken-eggs, sized around 2.7Gb.

Sam has to start moving eggs, so he just starts grabbing eggs and moving them. Let's say he decides to first move a dragon-egg. Now the egg is extremely massive and dense,so it takes him a while to pull the thing into the other basket.
Sam gets the dragon's egg into the other basket & he's pissed. That was hard because the thing took up so much space; the moving of it took way to long for just one single egg.

Sam is a fool.

Sam goes back to his 1st basket to grab another egg. He realizes he only really had 1 big one. There are however, 100,000,000,000 robbin's eggs left.

As you can see, Sam's afternoon is ruined. His natural inclination would be that the dino-egg should be way worse for him to have to move. That makes sense when we talk about eggs & baskets, but computers deal in numbers like 100,000,000,000.

In short:

The answer about why it takes so much longer to move several small files as compared to a few large ones, has to do with the cost associated with the action of moving them. The more small-files you have, the more times that action will need to be preformed. Another way of looking at this would be to talk about information density.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.