Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm looking for a slightly different kind of sync utility - not one designed to keep two directories identical, but rather one intended to keep files flowing from one host to another.

The context is a data warehouse that currently has a custom-developed solution that moves 10,000 files a day, some of which are 1+ gbytes gzipped files, between linux servers via ssh. Files are produced by the extract process, then moved to the transform server where a transform daemon is waiting to pick them up. The same process happens between transform & load. Once the files are moved they are typically archived on the source for a week, and the downstream process likewise moves them to temp then archive as it consumes them. So, my requirements & desires:

  • It is never used to refresh updated files - only used to deliver new files.
  • Because it's delivering files to downstream processes - it needs to rename the file once done so that a partial file doesn't get picked up.
  • In order to simplify recovery, it should keep a copy of the source files - but rename them or move them to another directory.
  • If the transfer fails (network down, file system full, permissions, file locked, etc), then it should retry periodically - and never fail in a non-recoverable way, or a way that sends the file twice or never sends the file.
  • Should be able to copy files to 2+ destinations.
  • Should have a consolidated log so that it's easy to find problems
  • Should have an optional checksum feature

Any recommendations? Can Unison do this well?

share|improve this question

closed as off-topic by Oliver Salzburg Sep 1 at 15:24

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they become outdated quickly and attract opinion-based answers. Instead, describe your situation and the specific problem you're trying to solve. Share your research. Here are a few suggestions on how to properly ask this type of question." – Oliver Salzburg
If this question can be reworded to fit the rules in the help center, please edit the question.

Unison is built on rsync. I would look for a system built on top of a message queue. Then each action (collect, transform, archive, forward) can be logged and repeated if necessary. Event driven. –  Jeremy W Jun 9 '12 at 18:14
Can the task be broken down into more simple steps? –  killermist Jun 11 '12 at 15:10
I've had to write this utility 3 times over the past fifteen years for different corporate employers or customers. And wouldn't mind writing an open source version if there's no good one yet (I've looked through alternativeto.net and can't find anything, but haven't combed through freshmeat, etc). I don't think you can really break this down much either: the process needs to look for qualifying files, move them, rename/move/etc source & dest after the move. –  KenFar Jun 11 '12 at 18:18
Have you looked at Oddjob, Open Source Job Scheduler, Quartz, and TORQUE ? –  Jeremy W Jun 11 '12 at 20:49
I hadn't looked at those yet, but just reviewed them now. Those are all schedulers - and aren't file transfer programs. They could be part of the solution (as could cron), but only provide the easy part. –  KenFar Jun 11 '12 at 23:35

1 Answer 1

I think that "sync" is the wrong word for what you're trying to do. In spite of this, I think rsync, properly tuned/controlled, would do what you're looking for.

Directories on each end could be used for incoming and outgoing files, and maybe have the same script that triggers the rsync copy do an SSH remote mv command on the completed block of files on the receiving machine from the incoming directory to a "ready to process" directory.

From the rsync man page:

These flags should make it copy once, with no duplicates, and allow graceful retry/resume:

     --partial               keep partially transferred files
     --remove-source-files   sender removes synchronized files (non-dirs)

The only problem with --remove-source-files is that it won't clean up empty directories after-the-fact (which I guess could be considered a feature, and not a flaw).

If using a nested directory structure,

 -r, --recursive             recurse into directories

I'm assuming that the ownership, groupship, etc. are not necessarily critical, but if they are, the following should solve that.

 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)

By default, it copies to a .hidden temp file of some sort and renames to the real name after complete. The following flag could be used if the files must be held segregated before complete.

 -T, --temp-dir=DIR          create temporary files in directory DIR
share|improve this answer
I agree that this isn't exactly a "sync" operation. But there doesn't appear to be a better common term for it. Also, I think if I do as you suggest that I'd need separate scripts to rename/move both the source & target side. This would take 2-3 configs, be difficult to make recoverable - especially since new files might be arriving while the process is running. I think I'd be better off writing something from scratch. –  KenFar Jun 12 '12 at 5:02
Oh. I kind of expected that you'd be writing something from scratch. For too many tasks, I find that mass-produced solutions are all poorly-fitted. –  killermist Jun 12 '12 at 5:12
Except - I've written utilities to do this several times, and once we had them the organization used them extensively. I've also seen quite a few organizations used Rsync for the same purpose, but in ways that are prone to losing or duplicating files delivered. –  KenFar Jun 12 '12 at 13:48
@KenFar Edited answer with (what I hope to be) more helpful pointers for rsync options/flags. Of course, I didn't see anything about sending to multiple targets. –  killermist Jun 12 '12 at 16:31
This looks great - except that I'd rather not delete the source files - I'd prefer to move them to another directory. The reason is that then if the downstream server craters it's much easier to rebuild it to a point in time. –  KenFar Jun 14 '12 at 18:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.