Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 2 down vote favorite

I'm having a hard time getting my head around this.

My test setup has a shell script continuously calling 'ls -la' on a 1G file and printing out the time since the last time it ran. I then run a program to modify parts of the file and sync it to disk.

It doesn't matter whether I call fsync, or the system does a sync, or even if I use pwrite to write the different parts (still testing that bit), when the sync happens the 'ls -la' will freeze for the entire time of the sync - between 7-40 seconds (depending on the sparsity of the modifications).

If I use msync to sync chunks at a time, or try to fsync more frequently as I write, the duration get much larger (maybe 10x as long, but even longer depending on how frequently I do it). The msync above only writes at 16KB/Transaction, even if the pages are sequential.

I have read somewhere that OpenBSD implemented 'partial file writes' or something. I can't quite remember now.

Is there anyway I can do something similar with the efficiency of the fsync without the file gettings locked down for the whole time?

Actually, the 'A' problem (for which I think this 'B' is the solution), is to simply work with large files and 'encourage' them to get written to disk so that the memory can be freed quickly if it needs to be. Simply ommitting NO_SYNC is no good as the changes will happen at about the same time, causing this situation. None of the other madvise options seems to help either. That is, if I don't sync then the pages seem to stick around until I run low on memory where they will suddenly start to swap (albeit at only 16KB/Transaction and a very low MB/s).

How on earth do you work with large files on FreeBSD?

SOLUTION:

I found that by tuning my msync chunks and using MS_ASYNC instead of MS_SYNC in the msync call I can get the performance I want while still allowing other processes to open and mmap/read the file.

share|improve this question

migrated from stackoverflow.com Oct 16 '12 at 7:47

3 Answers

up vote 1 down vote

freeBSD will use free memory to cache disk I/O, as do other UNIXes. On a system with lots of free memory and few users, really large files can be completely parked in memory. So that it looks like more memory is used.

close() (fclose() ) and fsync (fflush() ) are the only system calls that force the OS to write the cache. This is true only if no other process has the file open. freeBSD does not have fdatasync which just writes cached data, but not metadata to physical disk.

From BSD 4.4 on You can track paging and file caching with the mincore() syscall.

So, you have to fflush after every few writes.

Play with disk caching parms:

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/configtuning-disk.html
share|improve this answer
Thanks for the links. I think my question is better asked with my second edit - I am quite happy for the whole file to be cached, that is essentially what I want - the problem is that if it is all cached then at some stage it needs to be written to disk, and when this happens the file is locked out for the whole time. The file should still be available for reading at any time. Flushing after every few writes means that the application is limited by disk, even if there is lots of memory available. – Haru Oct 17 '12 at 2:17
up vote 0 down vote

http://www.unix.com/man-page/FreeBSD/4/syncer/

Explains your problem clearly. The syncer flushes dirty buffers (updated cache) to disk periodically. Those 'periodic' flushes are what you want to avoid. See what sysctl can do for your problem.

share|improve this answer
The periodic flushes can be avoided by mapping with NO_SYNC. The problem is that at some stage the data will need to be sync'd to disk. When that happens, it does it all in one go and locks the file for the duration of the sync. In the case of large files this can easily be a few minutes. If I purposly sync very often so that the file isn't locked for as long - then the application is limited by disk speed even when there is lots of memory available. – Haru Oct 17 '12 at 3:04
You probably need SSD drives if this is actually the problem you make it out to be. Basically, from what you have said there IS no real solution to the wait. Now you indicate that your periodic sync slowed the process to disk I/O speeds, which has always been the case in the computer world. The answer is faster disk I/O, there is no other answer. – jim mcnamara Oct 17 '12 at 3:24
The problem is the os exclusive lock on the file when syncing. To show that disk io shouldn't be a problem - imagine having 2 identical mmaped files. Write the data to both of them. Have all reads happening on the one file. Sync the other file to disk - users can still access the non-syncing one as it isn't being synced. Once the first is synced, swap the pointers around and remove the temporary file. Application runs at memory speed regardless of the sync. Downside - complexity and using 2x memory! – Haru Oct 17 '12 at 3:41
Update - I've discovered the problem is actually only when opening (or mmaping) or the file while it is being synced - it doesn't affect the ability to read the mapping if the file is already opened and mapped. However this is still a problem for me. (Also hangs if calling 'read'/'pread'. I'm guessing any file-related system calls. – Haru Oct 17 '12 at 4:03
up vote 1 down vote

You are solving your problem (monitoring file's state) completely wrong. Instead of periodically rechecking the file's state (and running into I/O concurrency issues once in a while), your program should simply ask the kernel to be notified, when a particular file (or collection of files) change.

Mechanisms to do this exist on all modern Unixes, but, unfortunately, they aren't the same...

On BSD family of Unixes this is done with kqueue/kevent. On Linux there is inotify. On Solaris there is poll and /dev/poll.

There are cross-platform libraries, which hide the OS-implementation details and give you portable API. If you need portability, look for File Alterations Monitor or its more modern subset called gamin (ported in /usr/ports/devel/gamin). If your application is only for (Free)BSD, you can use kqueue/kevent directly.

share|improve this answer
I think you might have misunderstood my purpose for the polling - it was just to show that nothing else can open/read the file while it is being sync'd. What I want is to allow other processes (maybe not controlled by me) to be able to open up and read the data while it is being sync'd. I can do that now with msync using MS_ASYNC. But thank you - I do make use of kqueue/kevent for monitoring file changes etc in other areas. – Haru Oct 20 '12 at 1:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.