Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a bunch of files (between 500MB and 7GB) that I need uploaded to an S3 bucket. My connection is very flaky.

I tried uploading a 500MB file via s3cmd but it timed out after it was 91% done (which took 16 hours)

The I tried with CyberDuck, but the same happened. It failed after 20% or so, and when I tried to retry the transfer, it started over from the beginning. CyberDuck is supposed to have multipart support, but I guess not...

I could split the files up into smaller files like this: How do I split a .zip file into multiple segments?, but I'd rather not unless it was my only option. What is a good program that I can use that will allow me to upload big files to S3 with resume support?

share|improve this question

2 Answers 2

I just tried using s3tools (s3cmd-1.5.0-rc1). Got this hint from their FAQ http://s3tools.org/kb/item13.htm. See below.

Does s3cmd support multipart uploads?

Yes, the latest version of s3cmd supports Amazon S3 multipart uploads.

Multipart uploads are automatically used when a file to upload is larger than 15MB. In that case the file is split into multiple parts, with each part of 15MB in size (the last part can be smaller). Each part is then uploaded separately and then reconstructed at destination when the transfer is completed.

With this new feature, if an upload of a part fails, it can be restarted without affecting any of the other parts already uploaded.

There are two options related to multipart uploads in s3cmd. They are:

--disable-multipart

Disable multipart uploads for all files

and

--multipart-chunk-size-mb=SIZE

Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded-multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB.

So when I upload I choose the smallest chunk size. You should see below splitting and resuming of upload.

$ s3cmd put --multipart-chunk-size-mb=5 some_video.mp4 s3://some_bucket/

some_video.mp4 -> s3://some_bucket/some_video.mp4  [part 1 of 52, 5MB]
 5242880 of 5242880   100% in  164s    31.08 kB/s  done
some_video.mp4 -> s3://some_bucket/some_video.mp4  [part 2 of 52, 5MB]
 5242880 of 5242880   100% in  193s    26.46 kB/s  done
some_video.mp4 -> s3://some_bucket/some_video.mp4  [part 3 of 52, 5MB]
 2023424 of 5242880    38% in  135s    14.59 kB/s^CERROR: 
some_video.mp4' part 3 failed. Use
  /usr/local/bin/s3cmd abortmp s3://some_bucket/some_video.mp4 XXX_SOME_HASH_XXX
to abort the upload, or
  /usr/local/bin/s3cmd --upload-id XXX_SOME_HASH_XXX put ...
to continue the upload.
See ya!

Then I resume.

/usr/local/bin/s3cmd --upload-id XXX_SOME_HASH_XXX put --multipart-chunk-size-mb=5 some_video.mp4 s3://some_bucket/
share|improve this answer

I believe in Cyberduck in the transfers window, you can right click and select resume.

If that doesn't work, Cloudberry suports resuming uploads

share|improve this answer
    
For some reason in Cyberduck my resume does not work for S3 multi-part. Any hints? –  f01 Dec 18 '14 at 8:38

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.