Compressed disk image on Linux

Question

I just got my new computer with a much bigger harddisk. I think I copied all important files over but just to be sure, I'd like to keep a disk image of my old disk. To save space, I'd like to compress it but I didn't find an option to mount a compressed image.

My goals:

• Result must be easy to access
• No need to decompress the whole thing before I can access anything
• Files should be quick to locate - no TAR/CPIO archive
• Necessary space should be less than just copying the files over

So ideally, I'm looking for a read-only, compressed file system which I can create in a file and which grows automatically.

Answer 1

Use a forensic analysis software like GUYMAGER (open source, sourceforge.net). It has a nice UI which allows to quickly create a compressed disk image of an entire hard disk.

Use "Advanced forensic image (.aff)" This creates a single, compressed file (well, it also creates an .info file).

To modify the default compression rate 1 (fastest, but least compression). If you have a fast computer with lots of cores, you can change this by creating /etc/guymager/local.cfg:

AffCompression = 3

9 is the best but slowest compression. 3 gives a good compression with good performance.

Update

Mounting isn't as simple as it seems. First of all, you need AFFLIB (Ubuntu: aptitude install afflib-tools). Now you can get the raw disk image with [affuse][3] <image> <mount-point>

But for some reason, mounting the raw image fails. parted says the first partition starts with 1048576B but

mount -t ext4 -o loop,ro,offset=1048576 /mnt/backup.raw /mnt/backup

fails with the usual useless mount error:

mount: wrong fs type, bad option, bad superblock on /dev/loop0,
   missing codepage or helper program, or other error
   In some cases useful info is found in syslog - try
   dmesg | tail  or so

and dmesg says:

EXT4-fs (loop0): VFS: Can't find ext4 filesystem

Answer 2

As Matthias Krull suggests in the comment above, qcow2 can be a good format which supports compression to store a disk image in. It will be supported for quite a while as its the format for QEMU virtual machine disk images. Here's an example command that compresses /dev/sda to the compressed image file sda.qcow2. By default zlib compression is used, but I think zstd may have a better compression ratio. 8 threads are used to increase throughput, but you should change that to the number of processor cores you have.

qemu-img convert -c -p -S 512 -o compression_type=zstd" -f raw -O qcow2 -m 8 \
  /dev/sda sda.qcow2

Access the image as a readonly block device NBD, Network Block device, can be used, as follows:

modprobe nbd
qemu-nbd -r -c /dev/nbd0 sda.qcow2
kpartx -av /dev/nbd0

Then the file system can be mounted as normal using mount and the appropriate device in /dev/mapper.

Answer 3

For file systems that support block compression, one option is to use store the disk image directly as a file. I know that BTRFS supports this, but likely others too. In the case of BTRFS, you will likely need to enable the mount option compress-force to override the uncompressible heuristic. You'll also want to create a sparse file. This can be done with dd, as in this example:

dd if=/dev/sda of=/mount/btrfs/sda.img conv=sparse,fsync iflag=nocache oflag=nocache

This will create a sparse file, so zero sectors will not take up space, and BTRFS will compress blocks that are compressible (assuming you did turn that on). The compsize tool can be used to see how much the file is compressed. Using du will only show the size of the uncompressed file minus holes, so will not be accurate.

To mount any file systems in the image, the following can be used:

LOOPDEV=$(losetup --show -f /mount/btrfs/sda.img)
kpartx -av "$LOOPDEV"
mount /dev/mapper/"$LOOPDEV"p3 # To mount the 3rd partition

If you do not use a file system that has native compression, eg. ext4, you can still use this method, by wrapping the disk image in a BTRFS file system image. That could look like this:

truncate -s 100G backup-sda.btrfs.img
mkfs.btrfs backup-sda.btrfs.img
mount -o compress-force=zstd:9 backup-sda.btrfs.img /media/tmp
cd /media/tmp
# Now do commands to create disk image as above

This method allows more control over the compression algorithms and compress level than the method using qemu-img convert, which only allows the selection of compression method, not level, and has less available compression methods.