I am aggregating together data that is being generated at a bunch of disparate sources, in a mixed environment of various versions of windows (xp/7) and one linux (centos). The data is available via network shares. The data is being produced by different software at each location, creating and appending to files at varying intervals. Most of the data sources are closed systems I do not have administrative access to, which really complicates the situation.
I am needing to periodically attach to the network shares and collect together the data into a central location, used for further distribution. The data is being generated and collected 24/7, for weeks at a time. This will be run from a Windows 2008 server machine.
The only hassle I am having with organizing this, is copying a file from a remote source without locking the file. I had initially set up a system that worked well using robocopy, until my testing found that the data source software would fail if it tried to append to a file currently being copied.
Unfortunately using vss is not an option as they are not volumes that are shared.
So, I am after some way to copy the file that will simply not lock the file while it is being copied. I am handling the files by checking modify times and sizes before and after copy, and flagging those with differences. Not having the exact most recent files is not an issue, but data loss at the source is.
I have looked into various backup software solutions but all that I have tested still lock files when they can't use vss.
I don't want to have to implement my own file copy software for something that should be pretty simple.