Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a bunch of disk images, made with ddrescue, on an EXT partition, and I want to reduce their size without losing data, while still being mountable.

How can I fill the empty space in the image's filesystem with zeros, and then convert the file into a sparse file so this empty space is not actually stored on disk?

For example:

> du -s --si --apparent-size Jimage.image 
120G Jimage.image
> du -s --si Jimage.image 
121G Jimage.image

This actually only has 50G of real data on it, though, so the second measurement should be much smaller.

This supposedly will fill empty space with zeros:

cat /dev/zero > zero.file
rm zero.file

But if sparse files are handled transparently, it might actually create a sparse file without writing anything to the virtual disk, ironically preventing me from turning the virtual disk image into a sparse file itself. :) Does it?

Note: For some reason, sudo dd if=/dev/zero of=./zero.file works when cat does not on a mounted disk image.

share|improve this question
2  
Writing zeroes into a file will not create a sparse file. It's a different concept. As you seek/read a sparse file when the OS discovers the block of data isn't really there (the block list is empty for data in that region) it (the OS) auto magically fills the read buffer with zero bytes. –  hotei Jul 31 '10 at 20:27
add comment

5 Answers

up vote 7 down vote accepted

First of all, sparse files are only handled transparently if you seek, not if you write zeroes.

To make it more clear, the example from Wikipedia

dd if=/dev/zero of=sparse-file bs=1k count=0 seek=5120

does not write any zeroes, it will open the output file, seek (jump over) 5MB and then write zero zeroes (i. e. nothing at all). This command (not from Wikipedia)

dd if=/dev/zero of=sparse-file bs=1k count=5120

will write 5MB of zeroes and will not create a sparse file!

As a consequence, a file that is already non-sparse will not magically become sparse later.

Second, to make a file with lots of zeroes sparse, you have to cp it

cp --sparse=always original sparsefile

or you can use tar's or rsync's --sparse option as well.

share|improve this answer
1  
According to Wikipedia, writing zeros with dd will create a sparse file. Can you explain what "seeking" means? –  endolith Jul 31 '10 at 19:33
1  
What about cat then? There is nothing in the man page about sparse files, so I assume cat /dev/zero > zero.file is perfectly OK to fill empty space with zeros? –  Ludwig Weinzierl Aug 1 '10 at 8:03
2  
@endolith: Updated my answer to make clear what the difference is to use dd for writing zeroes or for seeking. –  mihi Aug 1 '10 at 11:54
2  
@Ludwig Weinzierl: Yes, that cat command will fill your entire disk (or at least the amount not reserved for root or by quotas) with "real" zeroes, and create no sparse files. –  mihi Aug 1 '10 at 11:58
1  
@endolith you will need extra space, yes. but since you can compress the tarball, you will only need space for the original file and a compressed version of the sparse file. –  mihi Jun 1 '11 at 16:16
show 4 more comments

Do you mean that your ddrescue created image is, say, 50 GB and in reality something much less would suffice?

If that's the case, couldn't you just first create a new image with dd:

dd if=/dev/zero of=some_image.img bs=1M count=20000

and then create a filesystem in it:

mkfsofyourchoice some_image.img

then just mount the image, and copy everything from the old image to new one? Would that work for you?

share|improve this answer
add comment

PartImage can create disk images that only store the used blocks of a filesystem, thus drastically reducing the required space by ignoring unused block. I don't think you can directly mount the resulting images, but going:

image -> partimage -> image -> cp --sparse=alway

Should produce what you want (might even be possible to stick the last step, haven't tried).

share|improve this answer
add comment

There's now a tool called virt-sparsify which will do this. It fills up the empty space with zeros and then copies the image to a sparse file. It requires installing a lot of dependencies, though.

share|improve this answer
add comment

I suspect you'll require a custom program written to that spec if that's REALLY what you want to do. But is it...?

If you've actually got lots of all-zero areas then any good compression tool will get it down significantly. And trying to write sparse files won't work in all cases. If I recall correctly, even sparse files take up a minimum of 1 block of output storage where the input block contains ANY bits that are non-zero. For instance - say you had a file that had an average of even 1 non-zero bit per 512 byte block - it can't be written "sparsely". By the way, you're not going to lose data if you compress the file with zip, bzip, bzip2 or p7zip. They aren't like mpeg or jpeg compression that is lossy.

On the other hand, if you need to do random seek reads into the file then compression might be more trouble than it's worth and you're back to the sparse write. A competent C or C++ programmer should be able to write something like that in an hour or less.

share|improve this answer
    
Interesting - a downvote yet I notice there's no refutation of what I wrote. If it's accurate but unhelpful that's not a reason to downvote. If it's not accurate and not helpful then it does deserve it. –  hotei Jul 31 '10 at 16:03
    
I see elsewhere that the OP had a question relating to mounting compressed images. I'm assuming this is a continuation of that thread. Knowing that I can now see why my suggestion of compression wasn't accepted. A simple C program is still an easy way to create sparse files. BUT - will the (unspecified) OS let you mount a sparse ISO. As picky as the Ubuntu ISO mounter is I'm not 100% sure that's going to work either... but best of luck in any case. –  hotei Jul 31 '10 at 16:35
2  
why reinvent the wheel? cp --sparse=always does the work fine –  mihi Jul 31 '10 at 18:36
    
@mihi: That's a good idea. I didn't know about the sparse option as it's not available in BSD flavors (freebsd.org/cgi/…) and I have never had the requirement to look at a Linux man page for cp (until today). –  hotei Jul 31 '10 at 20:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.