I've got a bit of a problem here...
I've got a render-farm full of machines that are writing to a central disk, and, sometimes, after it's written an image sequence (DPX files, 12Mb per frame), about a minute after each file is written, it suddenly drops from 12Mb to 4kb.
I'm pretty certain that there is nothing running that would be doing this. .
The application that is creating the files is Shake.
This problem has cropped up before, very occasionally. I've certainly never figured out what actually causes it.
The central disk is a 10Tb raid, which the machines connect to via NFS.
df -h gives me the following for it:
clarke:/Volumes/projects 8.9Ti 7.7Ti 1.1Ti 88% /mount/projects
So it's not like it's running out of space (I was at 700Gb free when it started having the problem tonight, and I've freed up a load more just in case that was something to do with it...
Any help would be much appreciated! I really want to figure out what's causing this, as it's certainly causing me a lot of pain...
Edit:
If you want more specs of the server:
- XServe running OSX Server 10.5.8
- Promise Raid VTrak E610f
Also: The data certainly was there, for a couple of minutes. I set off a process to copy it to another machine, and that other machine is seeing the images just fine.
More edit:
Is there some kind of tool that will track every single file creation or modification in a folder (or whole file system) and tell me exactly what's going on? I want to find out if there is some kind of rogue process doing something naughty (it works most of the time, so I suspect not) or something lower down than that...