2

On a server, I have a directory /opt/kafka/data/topics.

$ du -hs /opt/kafka/data/topics
52M     /opt/kafka/data/topics

When I tar this directory like

$ tar czfv /tmp/topics.tar.gz /opt/kafka/data/topics

I get a file size that makes sense

$ ls -alh /tmp/topics.tar.gz
-rw-r--r-- 1 user user  11M Jan 12 15:15 kafka

However, when I download topics.tar.gz to my local OS X computer and extract it, it occupies 10GB!


Upon examining the contents of /opt/kafka/data/topics on the server more closely, I noticed that according to ls it contains many 10MB files:

$ find /opt/kafka/data -type f -exec ls -alh {} \;
... [output]
-rw-r--r-- 1 user user 10M Jan 12 02:45 /opt/kafka/data/topics/user-entities-KTABLE-REDUCE-STATE-STORE-0000000178-changelog-1/00000000000000000000.index
-rw-r--r-- 1 user user 10M Jan 12 02:45 /opt/kafka/data/topics/user-entities-KSTREAM-KEY-SELECT-0000000123-repartition-2/00000000000000000012.index
... [and many more]

du reports that each of these 10MB files are 0 bytes:

$ du -h /opt/kafka/data/topics/user-entities-KTABLE-REDUCE-STATE-STORE-0000000178-changelog-1/00000000000000000000.index
0       /opt/kafka/data/topics/user-entities-KTABLE-REDUCE-STATE-STORE-0000000178-changelog-1/00000000000000000000.index

So, what is going on? Obviously I am missing something here:

  • du reports 52M total. This makes sense because the device that /opt/kafka/data is mounted on is only 5GBs, df reports that it's only 2% full and everything is still working.
  • tar gzips the contents to 10M. This makes sense too.
  • ls reports that many of the files are 10M on disk and when I extract the archive I get 10GBs.
  • du reports that each of these same files are 0 bytes.
  • mount reports that /dev/sdc on /opt/kafka/data type ext4 (rw,relatime,data=ordered)

Nothing adds up. Is there some kind of transparent on-disk compression I am not aware of?

  • Multiple hardlinks to the same file inside the source directory? Sparse files? – Kamil Maciorowski Jan 12 '18 at 16:39
  • @KamilMaciorowski no hardlinks; yes, the 10MB files (as reported by ls) are sparse. They are reported as 0 bytes by du and are indeed empty – Dmitry Minkovsky Jan 12 '18 at 16:40
  • How about tar --sparse option while creating the archive? Does it help? Link. – Kamil Maciorowski Jan 12 '18 at 16:44
  • Help with what? The problem isn't that the tar is large when I create it. The tar is quite small. The problem is that on-disk, these 10MB sparse files seem to occupy 0 bytes. How can that be? Is there some sort of compression at the filesystem level? Why does ls report 10MB but du reports 0 bytes? – Dmitry Minkovsky Jan 12 '18 at 16:45
  • Although... my man tar on OS X does include an option -S for extracting: (x mode only) Extract files as sparse files. For every block on disk, check first if it contains only NULL bytes and seek over it otherwise. This works similiar to the conv=sparse option of dd. This may be helpful. – Dmitry Minkovsky Jan 12 '18 at 16:48
5

Based on the discussion in the comments, all the files are sparse. This type of thing actually confuses a lot of people the first time they deal with it, so don't feel bad.

What's actually going on here with the values reported by ls and du?

This is most easily explained with an example.

Say you create an empty file, and then write 1MB of data to it starting right at the beginning. The resultant file will be 1MB in size, and take up 1MB on disk. Both ls and du will report the same 1MB size for the file.

Now say instead you create an empty file, and then call seek() to move 1MB into the file, and then write one byte. The resultant file will appear to be 1MB + 1 byte long, but there's only actually 1 byte of data in it.

On older filesystems, the second file would have taken a very long time to write that 1 byte of data, because the OS would be busy writing out 1MB of null bytes before it wrote out that final 1 byte of actual data.

This inefficiency (both in terms of time to create the file, and space used on disk) is where sparse files come in. Instead of writing out that 1MB of null bytes, an OS that supports sparse files (like all modern UNIX systems) will annotate in that filesystem's metadata that the region form 0-1MB is empty, and then only store that single byte you wrote. As a result, the file will appear to be 1MB + 1 byte long, but on disk it will only take up 1 byte. Additionally, when something goes to read that file, any regions the OS has annotated as empty will just read back as null bytes (so it looks no different to user programs from the first file).

This is where the discrepancy between the values reported by ls and du comes from. By default, ls reports the apparent size of files (that is, how much data would you read if you started reading the file at the first byte and read all the way to the end), while du reports the actual space used on disk by the file (usually not including other space saving tricks done by the OS like transparent compression). du agrees with df in this case because df only reports the amount of space that is actually physically being used on disk.

By changing that ls -l command to ls -ls, you will get an extra column showing the actual on-disk size for files, which should agree with du.

| improve this answer | |
  • I don’t feel bad, don’t worry ;) Thanks for pointing out the -s switch. I’ll look into that. – Dmitry Minkovsky Jan 12 '18 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.