1

First post, hope it helps someone else storing large amounts of data online.

I use a cloud service, Bitcasa (NOT recommended) to serve my media library and for personal backup. They've recently radically changed their business model, and given a 3 week notice of a 12x price increase part way in to a year long subscription on beta customers like myself who've been helping as paying customers for several years while they iterate their product. All this to say I recommend staying away from them. I've had it, will never trust them after breach of contract, so I'm in a rush trying to download and verify over 7TB of data.

Having already downloaded 7TB from Bitcasa, it appears some is corrupted. It's not clear whether the original data was compromised in original transfer, or in their database servers, or errors occurred in download. Courts ordered an extension to Nov 20th for customers to DL data, however I don't have long before the data must be DLed before the Bitcasa service change. Most of the data is DLed, but I don't have confidence in the transfer.

In short, what's the best fast, reliable way to verify TBs of online data against my DLed copy in a Mac Yosemite environment? My ISP has at best a 100/10 transfer rate. I'm not afraid of command line. File count is likely in the 100s of thousands, mix of large and small, .txt to GB size media. Given time constraints, the verification should be multithreaded if possible. Any suggestions welcome!

Cheers!

7
  • 2
    If you don't have the original data you have nothing to check it against.
    – Ramhound
    Nov 18, 2014 at 20:54
  • Have you contacted Bitcasa and explained that you need sufficient time to download all your data and verify that your download is not corrupted? Suggestion: if they are not helpful, then you could contact the court and explain politely that you request an extension to the deadline and give clear reasons why - if you can get other users to back up your request then that will add weight to your request. If there is any chance that you are breaching copyright on any of the data then do not approach the court, and just make sure that you have all your personal data. Nov 18, 2014 at 21:09
  • 2
    The trick of course is to generate hashes of the files that good before you need to check them. If you didn't generate any hashes there really is no way to know where the corruption happened after the fact.
    – Zoredache
    Nov 18, 2014 at 23:18
  • Ah, you were lucky to even get such notice. I did not :-( Indeed, NOT recommended. Did you get notified by email?
    – Arjan
    Nov 29, 2014 at 14:26
  • @Arjan... sorry to hear it. It was the worst managed transition, many lost data or were forced to pay.
    – thepen
    Dec 7, 2014 at 1:05

1 Answer 1

1

If command line is your friend then rsync might help you out. Just do not delete the 7TB you already copied. Make a backup of it if at all possible. It uses something called delta encoding to enable it to only transfer files or part of files that are different. This way you can be confident that you got what was on their servers.

I am not a Bitcasa user myself and can not test it the whole process but they seam to support rsync.

You will need to use the data you already have as a destination in order for this to work for you. The general syntax for using rsync is. The --progress option just gives you an idea how it is doing.

rsync --progress source_at_bitcasa your_7TB_copy

You can always just use some smaller subfolder to test it out before you run it on the whole data. You will very likely need to add a few more options depending on your setup.

I know it is generally a good practice to work an answers a bit longer but there are very many nice resources on rsync on-line. And I do not want to spend your time any longer on getting the initial post out.

2
  • Your answer is fine; We are not hear to write a manual for rsync. If your command does the job then its fine. You explain he might need to work with it a little also.
    – Ramhound
    Nov 19, 2014 at 0:53
  • thanks @user291035. I used terminal command ditto -V <source> <destination> but it wasn't multithreaded. eventually i found a great tool called BitcasaFileLister-master which offered exactly what I was looking for. now it's over and done, bitcasa hostageware deleted everyone's data that wasn't transferred. Fortunately I got the essential data down, thanks for everyone's suggestions.
    – thepen
    Dec 7, 2014 at 0:58

Not the answer you're looking for? Browse other questions tagged .