Contiguous physical allocation of a set of files in linux filesystem (ext4)

Question

I have a set of files that I wish to be allocated contiguously in the filesystem. I will be accessing all these files sequentially in a single read one after the other and I want to optimize reading them. I can't combine them into a single file , and I am looking for a solution which will allow them to be individual files.

I am using an ext4 file-system and I was wondering if there is some existing tool which might do this for me as I learnt that ext4 supports online block exchange and defragmentation. I tried using e4defrag on my directory of files , although it ensured each individual file was defragmented, each file itself was in a separate block not necessarily adjacent to the other files. (I used filefrag -v file_name to verify if they were being allocated next to each other or not)

EDIT: Just to clarify on file access patterns, These files will be written exactly once and never modified again. They will be read frequently, but in such a manner that if any one of them is read all the other files in the set will also likely be needed to be read. What I intend to do is prefetch all these files together into the filesystem buffer/cache in one go, so that subsequent random reads of any of these files will be really fast. (The total file size is small enough 100~200MB to fit in cache). What I am trying to improve right now is the read performance when I try to load these files into cache. Right now read performance takes a hit while trying to prefetch them into cache because there are multiple disk seeks as they are located in disjoint segments.

Answer 1

Not exactly a way to re-organize filesystem blocks, but…

You want to have the files in RAM and you said in comments you had already experimented with ramdisk. We can improve this approach.

My idea is to read files not directly from a filesystem like ext4 but from a .tar file. You would create this file once, place it on ext4 filesystem and defragment with e4defrag. Then at every boot

cd /mnt/target_tmpfs/ &&  tar -xf /mnt/ext4/defragmented_archive.tar

I don't think tar will look around in the given file, seek back and forth. But if you do think so then you can always use cat … | tar -x. In this case tar can only read its input in a continuous manner and at most once.

I'm aware you cannot easily load entire OS this way, unless maybe you prepare initramfs to do it. I don't know much about it but I've found this: Custom Initramfs. From therein:

Here are some examples of what you can do with initramfs:

• Mount the root partition (for encrypted, logical, and otherwise special partitions);
• […]

See example scripts there. Mounting ext4, then tmpfs and populating it from .tar, then using this as / – it all seems possible in general.

Obviously you would like your custom-initramfs.cpio.gz to be defragmented as well as the .tar file.

Answer 2

rsync can write files contiguously:

rsync --preallocate /path/to/source/file /path/to/destination/

rysnc preallocates a contiguous block of storage and copies the file into it. Works for FAT too.

Just make sure the file does not already exist at the destination, or rsync won't reallocate and re-copy it. If it is, delete it, Empty Trash to make sure it's really gone, then run this command.

Verify if it copied contiguously:

filefrag /path/to/destination/file

"1 extent found" means the file is contiguous. More than one means it's fragmented.

Doing what you need may require a bit more scripting, but should be doable with this command.