I'm having a hard time getting my head around this.
My test setup has a shell script continuously calling 'ls -la' on a 1G file and printing out the time since the last time it ran. I then run a program to modify parts of the file and sync it to disk.
It doesn't matter whether I call fsync, or the system does a sync, or even if I use pwrite to write the different parts (still testing that bit), when the sync happens the 'ls -la' will freeze for the entire time of the sync - between 7-40 seconds (depending on the sparsity of the modifications).
If I use msync to sync chunks at a time, or try to fsync more frequently as I write, the duration get much larger (maybe 10x as long, but even longer depending on how frequently I do it). The msync above only writes at 16KB/Transaction, even if the pages are sequential.
I have read somewhere that OpenBSD implemented 'partial file writes' or something. I can't quite remember now.
Is there anyway I can do something similar with the efficiency of the fsync without the file gettings locked down for the whole time?
Actually, the 'A' problem (for which I think this 'B' is the solution), is to simply work with large files and 'encourage' them to get written to disk so that the memory can be freed quickly if it needs to be. Simply ommitting NO_SYNC is no good as the changes will happen at about the same time, causing this situation. None of the other madvise options seems to help either. That is, if I don't sync then the pages seem to stick around until I run low on memory where they will suddenly start to swap (albeit at only 16KB/Transaction and a very low MB/s).
How on earth do you work with large files on FreeBSD?
I found that by tuning my msync chunks and using MS_ASYNC instead of MS_SYNC in the msync call I can get the performance I want while still allowing other processes to open and mmap/read the file.