I'm working with an academic project that's trying to optimize energy efficiency for homeowners by collecting sensor data in smart homes. The problem is that they have to upload their sensor data from a network of heterogeneous patchwork-quilt machines (some Windows laptops, some Linux servers, some Raspberry Pis, etc.).
The solution needs to have the following properties:
Sync to/from S3. We've already got some data on S3, but we don't really care about the provider as long as it's cheap and low-failure backing storage that looks like a directory.
One-way, per-file sync. It should be able to upload changes for only a specific file, rather than for an entire directory. It should also be able to push changes.
Delta writes. The files will be large and we don't want to re-upload them in full each time -- only the changes.
Stores metadata separately. We need to be able to read the files on the remote end as they are on the local end; if the files are wrapped in some kind of envelope/container that will make it harder to consume them.
Command-line interface. It needs to be scriptable because we'll be driving the synchronization process with an external script.
Cross-platform. It should work on Windows 7+ and Ubuntu.
Minimal external dependencies. Ideally, "installation" would consist of dropping a folder somewhere or downloading a package.
Options I've looked at and had to reject:
Write directly using s3fs: Doesn't support delta writes; re-uploads entire file for any change. Ouch!
rsync + s3rsync.com: Prohibitively expensive, since our rsync streams will be uploading lots of data, all the time. Also, site is defunct ("copyright 2010"), lots of bad reviews.
Duplicity: Doesn't support writing raw file content -- every file is wrapped in metadata that only Duplicity understands.
Syncany: Doesn't support one-way, single-file sync.
