Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have several copies of a file. The files take 2390170KiB each, according to ls -lk and du --apparent-size -k.

The problem is that du -k reports different sizes for each copy: 2389824, 2392512, 2392512 and 2390336.

I expected the disk usage should be the 2390172KiB (the size of the minimun number of blocks (597543) where the file fits.

So, why each file have a different disk usage?

I also have seen that one of the copies use 2389824KiB, but the file size is 346KiB bigger: 2390170. How does this make sense?

PD: All the files are in the same ext3 filesystem. The filesystem block size is 4096. All the files have the same hash.

Update: From the comments:

although the apparent size is usually smaller, it may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like

sparse files may be the reason because the disk usage is lower. But I fail to see how internal fragmentation or indirect blocks may reduce the disk usage in respect to the original file. Since the file is the same, the disk usage from the internal fragmentation and the indirect blocks should be constant.

I have observed that cp --sparse=always can make a sparse file from a non-sparse one. cp --sparse=always results on a file that uses 2390336KiB cp --sparse=never results on a file that uses 2392512KiB

So I'll guess that the 2389824KiB usage from one of the copies is caused from a different implementation of the sparse algorithm...

The original file was copied from a windows machine through sftp or samba, and I think that the 2389824KiB file is a copy of it, but I don't remember how did I did it (I guess that with cp, but I'm not sure).

share|improve this question
add comment

2 Answers

From man du:

--apparent-size
print apparent sizes, rather than disk usage; although the apparent size is usually smaller, it may be larger due to holes in ('sparse') files, internal fragmentation, indirect blocks, and the like

From info du:

'--apparent-size'
Print apparent sizes, rather than disk usage. The apparent size of a file is the number of bytes reported by wc -c on regular files, or more generally, ls -l --block-size=1 or stat --format=%s. For example, a file containing the word 'zoo' with no newline would, of course, have an apparent size of 3. Such a small file may require anywhere from 0 to 16 KiB or more of disk space, depending on the type and configuration of the file system on which the file resides. However, a sparse file created with this command:

      dd bs=1 seek=2GiB if=/dev/null of=big

has an apparent size of 2 GiB, yet on most modern systems, it actually uses almost no disk space.

[emphasis mine]

share|improve this answer
add comment

There are two things going on here -- the files that are larger than you expect are due to metadata the filesystem stores that is being counted towards the file's size. The files that are smaller than you expect are due to a UNIX feature called "sparse files".

Larger Files

For ext2/ext3, the on-disk usage of the file includes the space used up by the filesystem structures that keep track of where the data blocks are on the disk. Take a look at the Ext2 inode structure -- an inode is the data structure that keeps track of a file's permissions, size, etc. as well as where its data blocks are on disk. The inode itself isn't counted towards the usage (it's preallocated at filesystem creation), but the indirect blocks are.

Calculation

So, your file with size 2390172kB takes 597543 blocks of data, like you said. The locations of 12 of those blocks are stored in the inode itself, so they're free. The 13th location stored in the inode is for an indirect block -- a newly-allocated block that stores the locations of 1024 data blocks. So that adds 1 block to your filesize, and leaves us with 596507 blocks.

The 14th location pointer in the inode is for a doubly indirect block -- a block allocated that contains space the locations of 1024 indirect blocks. 596507/1024 ~= 582.52, so we'll need 583 indirect blocks to contain the rest of the data blocks, plus the one doubly-indirect block.

So:

    1 (indirect from inode)
+   1 (doubly-indirect from inode)
+ 583 (indirect from doubly-indirect)
-----
= 585
= 598128 - 597543

And that accounts for the size 2392512 (=598128*4).

Smaller Files

I suspect that the smaller files (2389824kB) are sparse files, which means that some of the blocks were never written to and so weren't allocated -- unallocated blocks are defined as being filled with zeroes. See Dennis Williamson's answer for references. Sparse files can occur if the program writing moves the file pointer around and writes to different positions in the file, instead of writing the file through from beginning to end. For an extreme example of a sparse file, try the following:

du if=/dev/zero of=my_sparse_file bs=1000 count=1 seek=1000000

If you ls the resulting file, the apparent size will be 1000001000. However, because only 1000 bytes were written, only one data block is used, so only one data block is allocated. du will report 12kB used -- one 4k block for the data, one for the doubly-indirect block, and one for a single indirect block that the doubly-indirect block points to with its 976th pointer. None of the rest of the file's blocks were allocated, whether data or metadata.

Once the doubly-indirect block runs out, the filesystem starts using a triply indirect block. Your file will hit the maximum possible ext3 filesystem size before it fills that one.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.