I am aggregating together data that is being generated at a bunch of disparate sources, in a mixed environment of various versions of windows (xp/7) and one linux (centos). The data is available via network shares. The data is being produced by different software at each location, creating and appending to files at varying intervals. Most of the data sources are closed systems I do not have administrative access to, which really complicates the situation.

I am needing to periodically attach to the network shares and collect together the data into a central location, used for further distribution. The data is being generated and collected 24/7, for weeks at a time. This will be run from a Windows 2008 server machine.

The only hassle I am having with organizing this, is copying a file from a remote source without locking the file. I had initially set up a system that worked well using robocopy, until my testing found that the data source software would fail if it tried to append to a file currently being copied.

Unfortunately using vss is not an option as they are not volumes that are shared.

So, I am after some way to copy the file that will simply not lock the file while it is being copied. I am handling the files by checking modify times and sizes before and after copy, and flagging those with differences. Not having the exact most recent files is not an issue, but data loss at the source is.

I have looked into various backup software solutions but all that I have tested still lock files when they can't use vss.

I don't want to have to implement my own file copy software for something that should be pretty simple.

  • This is a tricky job, if you want an atomic view. It's done either by making a shadow copy (a la Unix) or by enabling a journal on the file. But if you can tolerate a changing view then it's a simple matter of opening a shared read that allows simultaneous write (if the OS permits that mode). – Daniel R Hicks Feb 10 '14 at 23:01
up vote 2 down vote accepted

Not sure if XXCOPY can do what you want so you may want to read a post regarding its access to locked files. Otherwise, it sounds like XCOPY (and RoboCopy) on steroids. Not free for business use, although it is still being supported (latest version released on 12/27/2013)

Addition: The documentation says that XXCOPY itself does not lock the file while it copies it.

  • Great, thanks for that - XXCOPY looks like a viable way to go. the flag /SHRW opens in shared mode, which is just what I am looking for. Just tested it now, and my acquisition systems are able to continue appending to the data file during the copy :) – Steve Feb 10 '14 at 23:18
  • Deserves a +1 @JSanchez, good find. I've used it in the past and had completely lost track of it. – Julian Knight Feb 11 '14 at 8:14
  • @JulianKnight: Thanks! I had used it a long time ago, when I used to work on this DOS-based POS and the XCopy command wasn't cutting it for what I needed to do. Glad to know it's still under development, and that Steve was able to use it. My Boy Scout deed for the week. ;-) – JSanchez Feb 11 '14 at 22:47
  • Thanks @JSanchez, XXCopy does do what I want. The only sticking point is their licensing, which requires a license for every machine that xxcopy touches. This might be problematic as a portion of the machines that make up the environment would belong to external visiting groups, rotating through every month or so. That however is an issue separate to what I outlined. Thanks again! – Steve Feb 11 '14 at 23:42

FileCopy.exe claims to be able to do this but it is rather old (2004) so I don't know if it will work for you. In addition, it isn't clear whether the copy would remain non-blocking when used over a network share.

It is, of course, quite dangerous to do non-blocking copies which is why most utilities don't support it. If the master application updates the file while you are copying, you are likely to end up with a corrupted copy.

  • Non-blocking is a rather different concept when it comes to opening files. The right terminology is sharing mode, in this particular case with a sharing mode that allows reading and writing (and possibly FILE_SHARE_DELETE as well, although the doco indicates probably not) by other processes that open the file. – JdeBP Feb 7 '14 at 13:51
  • FileCopy does look a bit old, but I will still check it out, thanks. I can live with receiving/handling a corrupted file, as long as I can ensure that there won't be any loss of data at the source. Thanks for the clarification on the terminology. – Steve Feb 10 '14 at 23:13
  • Also, he says he is “checking modify times and sizes before and after copy, and flagging those with differences” -- that should detect most copies that are corrupted by concurrent modification of the source file. – Scott Feb 11 '14 at 1:25

Just to add another option, I have found FastCopy to copy files in shared read/write mode.

It seems fairly mature and has been around since 2004, and is open source under the BSD license found here. It is seeming like a good alternative, as not all the extra bells and whistles that XXCOPY provides are required.

try shadowspawn (with robocopy). It is quite good.

shadowspawn

Edit: i just realized that this is 2 years old. Sorry!

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.