I have a DD image from a 4GB SD card that has two partitions, these two partitions are only using up about 800 MB and as such I wish to reduce the size of the img fie.
Does anyone know of a way to remove the "free space" from the img file?
Super User is a question and answer site for computer enthusiasts and power users. It only takes a minute to sign up.Sign up to join this community
First make sure the free space is actually empty, and doesn't contain leftovers of deleted files.
With recent kernels (3.2 or later), it's easiest to do so by mounting each partition of the loop image, then issuing a discard using
fstrim on the mountpoint. This works on loop devices in a similar way to TRIM on SSDs; unused areas are replaced with zeros and the underyling
.img file becomes sparse.
# losetup --find --partscan foo.img # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 4096M 0 loop ├─loop0p1 259:0 0 2048M 0 loop └─loop0p2 259:1 0 2048M 0 loop # for part in /dev/loop0p*; do mount $part /mnt fstrim -v /mnt umount /mnt done /mnt: 2xxx MiB trimmed /mnt: 2xxx MiB trimmed # losetup --detach /dev/loop0
Otherwise, an easy way to achieve this is to create a huge file on the disk, containing only null bytes, then delete it.
# losetup --find --partscan foo.img # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 4096M 0 loop ├─loop0p1 259:0 0 2048M 0 loop └─loop0p2 259:1 0 2048M 0 loop # for part in /dev/loop0p*; do mount $part /mnt dd if=/dev/zero of=/mnt/filler conv=fsync bs=1M rm /mnt/filler umount /mnt done dd: error writing ‘/mnt/filler’: No space left on device dd: error writing ‘/mnt/filler’: No space left on device # losetup --detach /dev/loop0
Then compress it with a tool like
xz. Even at lowest compression levels, a long series of zeros will compress well:
# ls -s 4096M foo.img # gzip foo.img # ls -s 11M foo.img.gz
Note that you must uncompress the image when writing it back to disk. This will uncompress it 'live':
# cat foo.img.gz | gunzip | dd of=/dev/sda
Note that the output device (sda) must be of sufficient size to fit the original image, otherwise data will be lost or corrupted.
An alternative method, if you want to keep using the image – e.g. with a virtual machine – is to convert the raw image to one of the image formats used by virtualization software; e.g. qcow2 for Qemu, VDI for VirtualBox, or VMDK for VMware.
Note that this still requires you to prepare the image by cleaning the free space using the above method.
# qemu-img convert -f raw -O qcow2 foo.img foo.qcow # qemu-img convert -f raw -O vmdk foo.img foo.vmdk
But if it's going to be written to a real disk again, you have to convert it back to a raw image.
I originally posted the same answer here, on StackExchange Ask Ubuntu. I re-propose the same answer here, it can be useful.
The key information was the use of the command
truncate. Following the full solution in order to not lose the answer.
A preliminary step consists in cloning the SD card in your PC:
lsblk to see which devices are available and if their partitions are mounted
unmount all partitions of the device you want to copy on your pc. For example:
umount /dev/sdc1 umount /dev/sdc2
create a copy of the whole sd card with all the partitions unmounted
dd if=/dev/sdc of=/path/to/file/myimage.img
Context of the problem:
myimage.img bigger then the hardware support (if it is smaller there should be no problem; however, using the same strategy, you can better fit the image in the hardware support).
The secret is to use standard Linux tools and instruments: GParted,
.imgyou want to shrink (
myimage.imgin this example)
Creating loopback device:
GParted is an application typically used to manage partition tables and filesystems. In order to shrink the image, GParted is going to be used along the first part of the answer.
GParted operates on devices, not simple files like images. This is why we first need to create a device for the image. We do this using the loopback-functionality of Linux.
Let's enable enable the loopback:
sudo modprobe loop
Let's request a new (free) loopback device:
sudo losetup -f
The command returns the path to a free loopback device:
Let's create a device of the image:
sudo losetup /dev/loop0 myimage.img
myimage.img. We want to access the partitions that are on the image, so we need to ask the kernel to load those too:
sudo partprobe /dev/loop0
This should give us the device
/dev/loop0p1, which represents the first partition in
myimage.img. We do not need this device directly, but GParted requires it.
Resize partition using GParted:
Let's load the new device using GParted:
sudo gparted /dev/loop0
When the GParted application opens, it should appear a window similar to the following:
Now notice a few things:
We want to resize this partition so that is fits its content, but not more than that.
Select the partition and click Resize/Move. A window similar to the following will pop up:
Drag the right bar to the left as much as possible.
Note that sometimes GParted will need a few MB extra to place some filesystem-related data. You can press the up-arrow at the New size-box a few times to do so. For example, I pressed it 10 times (=10MiB) for FAT32 to work. For NTFS you might not need to at all.
Finally press Resize/Move. You will return to the GParted window. This time it will look similar to the following:
Notice that there is a part of the disk unallocated. This part of the disk will not be used by the partition, so we can shave this part off of the image later. GParted is a tool for disks, so it doesn't shrink images, only partitions, we have to do the shrinking of the image ourselves.
Press Apply in GParted. It will now move files and finally shrink the partition, so it can take a minute or two, but most of the time it finishes quickly. Afterwards close GParted.
Now we don't need the loopback-device anymore, so unload it:
sudo losetup -d /dev/loop0
Shaving the image:
Now that we have all the important data at the beginning of the image it is time to shave off that unallocated part. We will first need to know where our partition ends and where the unallocated part begins. We do this using
fdisk -l myimage.img
Here we will see an output similar to the following:
Disk myimage.img: 6144 MB, 6144000000 bytes, 12000000 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000ea37d Device Boot Start End Blocks Id System myimage.img1 2048 9181183 4589568 b W95 FAT32
Note two things in the output:
1 * 512)
We will use these numbers in the rest of the example. The block-size (512) is often the same, but the ending block (9181183) will differ for you. The numbers mean that the partition ends on byte 9181183512 of the file. After that byte comes the unallocated-part. Only the first 9181183512 bytes will be useful for our image.
Next we shrink the image-file to a size that can just contain the partition. For this we will use the
truncate command (thanks uggla!). With the truncate command need to supply the size of the file in bytes. The last block was 9181183 and block-numbers start at 0. That means we need (9181183+1)*512 bytes. This is important, else the partition will not fit the image. So now we use truncate with the calculations:
truncate --size=$[(9181183+1)*512] myimage.img
I have used the gparted approach with my Ubuntu 16.10 computer:
Map the img file to the next available loop partition with
sudo losetup -f --partscan file.img
lsblk which loop drive your image file is mapped to, e.g.
sudo gparted /dev/loop0
Shrink the loop partition(s) as deemed appropriate; please make sure to have these partitions unmounted.
fdisk /dev/loop0, then enter
p, this will show you the block size and end block number of the various partitions.
sudo dd if=/dev/loop0 of=shrunk_image_file.img, apply to that command the options
count=[EndBlockNumberOfLastLoopPartition+1] and you will have a shrunk and rightsized image file. Optionally add
status=progress to see it working.
When done unmap the image file with
sudo losetup -d loop0
If you are working on a raspberry pi image, then you can try pishrink: it's very easy to use and minimize your pi image file automatically.
Looked through many of the methods here and I understood all of them but the steps were something I wasn't looking forward to do in having to execute them in turn to simply shrink a Raspberry Pi IMG file down. So looking I came across this Bash script from a project called PiShrink which was mentioned by @huan here in his answer. I took his suggestion and put a full flow around using PiShrink.
NOTE: It was quite trivial to execute and it worked on my MacOS M1 Mac Mini to boot.
In my particular case I had 2 32GB SD cards and for whatever reason 1 was slightly (~95MB) smaller than the other and balenaEtcher refused to write the IMG from what seemed to be the larger of the 2 SD cards to the smaller.
I knew I was only using ~10GB of the 32GB so I could shrink the IMG file down to fit.
To start I had the following files. The
.zip file contained the
.img file but I'm showing it unzipped from it below.
$ ls -lh | grep -E 'img|zip' -rw-r--r-- 1 root staff 28G Jan 29 13:16 backup.zip --w--wx-w- 1 slm staff 30G Jan 1 1980 disk.img
I then downloaded the PiShrink shell script and made it executable:
$ wget https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh $ chmod +x pishrink.sh
I then ran it like so:
$ sudo pishrink disk.img pi.img Copying disk.img to pi.img... e2fsck 1.44.0 (7-Mar-2018) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information STORAGE: 69402/7684096 files (0.2% non-contiguous), 9562823/30732288 blocks resize2fs 1.44.0 (7-Mar-2018) resize2fs 1.44.0 (7-Mar-2018) Resizing the filesystem on /dev/disk4s2 to 9273700 (1k) blocks. Begin pass 2 (max = 245358) Relocating blocks XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Begin pass 3 (max = 3752) Scanning inode table XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Begin pass 4 (max = 3420) Updating inode references XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX The filesystem on /dev/disk4s2 is now 9273700 (1k) blocks long. "disk4" ejected. fdisk: could not open MBR file /usr/standalone/i386/boot0: No such file or directory Enter 'help' for information fdisk: 1> Starting Ending #: id cyl hd sec - cyl hd sec [ start - size] ------------------------------------------------------------------------ 2: 83 1023 3 32 - 1023 254 2 [ 1056768 - 61464576] Linux files* Partition id ('0' to disable) [0 - FF]:  (? for help) Do you wish to edit in CHS mode? [n] Partition offset [0 - 62521344]:  Partition size [1 - 61464576]:  fdisk:*1> Writing MBR at offset 0. fdisk: 1> Shrunk pi.img from 30G to 9.3G
The resulting IMG file is now ~9GBs.
$ ls -lh | grep -E 'img|zip' -rw-r--r-- 1 root staff 28G Jan 29 13:16 backup.zip --w--wx-w- 1 slm staff 30G Jan 1 1980 disk.img --w---x--- 1 root staff 9.4G Jan 29 17:04 pi.img
I then repackaged it as a ZIP file just so that it was kept in the most efficient size possible while dealing with it on disk.
$ sudo zip pi.zip pi.img
We now have the following set of files kicking around:
$ ls -lh | grep -E 'img|zip' -rw-r--r-- 1 root staff 28G Jan 29 13:16 backup.zip --w--wx-w- 1 slm staff 30G Jan 1 1980 disk.img --w---x--- 1 root staff 9.4G Jan 29 17:04 pi.img -rw-r--r-- 1 root staff 8.3G Jan 29 17:08 pi.zip
Poking at the SD card on my Mac I can see it's showing ~10GB.
$ diskutil list /dev/disk4 /dev/disk4 (external, physical): #: TYPE NAME SIZE IDENTIFIER 0: FDisk_partition_scheme *31.9 GB disk4 1: Windows_FAT_32 LIBREELEC 536.9 MB disk4s1 2: Linux 10.0 GB disk4s2 (free space) 21.3 GB -
NOTE: I could do the same commands from a Linux system using
I did notice after using the above method that the filesystem within the SD card was short on the full 32GB. Mainly it showed like so:
$ df -h | grep -E "File|mmc" Filesystem Size Used Available Use% Mounted on /dev/mmcblk0p1 511.7M 141.7M 370.0M 28% /flash /dev/mmcblk0p2 8.6G 8.2G 377.4M 96% /storage
This should be easily resolved by expanding the SD filesystem from within once you've booted it back up.
To start this I first use
parted to extend the partition:
$ parted /dev/mmcblk0 resizepart 2 100% Warning: Partition /dev/mmcblk0p2 is being used. Are you sure you want to continue? Yes/No? yes yes Information: You may need to update /etc/fstab.
I then rebooted the system:
After I then online expanded the filesystem using
$ resize2fs /dev/mmcblk0p2 resize2fs 1.45.6 (20-Mar-2020) Filesystem at /dev/mmcblk0p2 is mounted on /storage; on-line resizing required old_desc_blocks = 71, new_desc_blocks = 234 The filesystem on /dev/mmcblk0p2 is now 30638592 (1k) blocks long.
And it now shows all the free space on the SD card:
$ df -h | grep -E "File|mmc" Filesystem Size Used Available Use% Mounted on /dev/mmcblk0p1 511.7M 141.7M 370.0M 28% /flash /dev/mmcblk0p2 28.3G 8.2G 20.1G 29% /storage