11

How do I remove all holes in a sparse file on Linux? On Windows, I believe the procedure is

fsutil sparse setflag <file> 0

What is the Linux equivalent?

(I know cp <file> <file>.cp; rm <file>; mv <file>.cp <file>; would get the job done, if free disk space permits, but I guess Linux has a in-place way of doing it? Like Windows?)

2
  • 5
    Can I ask what it is you're trying to achieve?
    – marcelm
    Dec 10, 2023 at 21:36
  • 1
    @marcelm just curious. It started when I made a swap file with truncate -s 4G swap.bin; mkswap swap.bin; and mkswap informed me that the file was sparse and the kernel might refuse to mount it. I know i could convert it to non-sparse with dd if=/dev/zero, but it made me wonder what the general way of converting a file to non-sparse is in Linux. I asked ChatGPT3.5, it didn't know. First few google/superuser search results didn't answer the question either. So I asked :)
    – hanshenrik
    Dec 13, 2023 at 15:44

1 Answer 1

16

fallocate -l <length> <file> might achieve this; while it doesn't tell the filesystem to explicitly zero-fill the holes, it does allocate space for them. Compare with:

[make a sparse file]
$ echo yay | dd bs=1 seek=1G of=test.file

[list extents]
$ filefrag -v test.file
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:   262144..  262144:  902523324.. 902523324:      1:     262144: last,eof

[allocate space]
$ fallocate -l $(stat -c %s test.file) test.file

[list extents again]
$ filefrag -v test.file
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..  196607: 2219092376..2219288983: 196608:             unwritten
   1:   196608..  262143: 2219301264..2219366799:  65536: 2219288984: unwritten
   2:   262144..  262144:  902523324.. 902523324:      1: 2219366800: last,eof

Use filefrag -v <file> or xfs_io -r -c "fiemap -v" <file> to list a file's extents. In this example, after fallocate there are no more holes as such; they're now covered by 'unwritten' extents that the filesystem will continue to immediately return zeros on read, but which now do have specific physical blocks assigned to them.

It would be technically possible to add an option to fallocate which uses FIEMAP to list holes and/or "unwritten" extents and zero-fill them, but unfortunately it doesn't have one at the moment and requires the zero-fill offset/length to be specified by hand.


  • If the file happens to get truncated in between the stat -c %s and the fallocate, the latter operation will grow it again to its previous size.

    You might be able to avoid this using -n/--keep-size (which only pre-allocates space but doesn't grow the file) if you expect it to grow again, though the exact Linux semantics of allocating space beyond EOF are a bit unclear to me. (I suppose it's similar to the Windows 'fsutil file setEOF/setValidData'.)

    On the other hand, if the file grew in between the two operations, it will not be truncated; the newly added data (or holes) will just remain unaffected by the fallocate. (That is, you will not lose any data; worst case you'll just need to do it again to get rid of newly added holes.)

  • Be careful with cp – in recent GNU Coreutils versions, it will automatically try to preserve sparseness using SEEK_HOLE. You might need to use cp --sparse=never to prevent this.

  • In addition, as of Coreutils 9.x, cp foo bar and even cat foo > bar will automatically use the FICLONE or copy_file_range() operations to clone files without actually copying their data if the filesystem supports this, e.g. on XFS or Btrfs or ZFS (similar to the "block cloning" feature in ReFS).

    If this happens, the "copy" will exactly retain the original sparseness of the file, and you will see the shared flag for each extent in filefrag's output. You can use cp --reflink=never to prevent this.

    (There is a flag for the fallocate system call to unshare extents, but it hasn't yet been exposed through the fallocate command-line tool, although you can do it through xfs_io which has the 'funshare' subcommand.)

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .