How to upload big files to S3 on a flaky connection?

Question

I have a bunch of files (between 500MB and 7GB) that I need uploaded to an S3 bucket. My connection is very flaky.

I tried uploading a 500MB file via s3cmd but it timed out after it was 91% done (which took 16 hours)

The I tried with CyberDuck, but the same happened. It failed after 20% or so, and when I tried to retry the transfer, it started over from the beginning. CyberDuck is supposed to have multipart support, but I guess not...

I could split the files up into smaller files like this: How do I split a .zip file into multiple segments?, but I'd rather not unless it was my only option. What is a good program that I can use that will allow me to upload big files to S3 with resume support?

Answer 1

I just tried using s3tools (s3cmd-1.5.0-rc1). Got this hint from their FAQ http://s3tools.org/kb/item13.htm. See below.

Does s3cmd support multipart uploads?

Yes, the latest version of s3cmd supports Amazon S3 multipart uploads.

Multipart uploads are automatically used when a file to upload is larger than 15MB. In that case the file is split into multiple parts, with each part of 15MB in size (the last part can be smaller). Each part is then uploaded separately and then reconstructed at destination when the transfer is completed.

With this new feature, if an upload of a part fails, it can be restarted without affecting any of the other parts already uploaded.

There are two options related to multipart uploads in s3cmd. They are:

--disable-multipart

Disable multipart uploads for all files

and

--multipart-chunk-size-mb=SIZE

Size of each chunk of a multipart upload. Files bigger than SIZE are automatically uploaded as multithreaded-multipart, smaller files are uploaded using the traditional method. SIZE is in Mega-Bytes, default chunk size is 15MB, minimum allowed chunk size is 5MB, maximum is 5GB.

So when I upload I choose the smallest chunk size. You should see below splitting and resuming of upload.

$ s3cmd put --multipart-chunk-size-mb=5 some_video.mp4 s3://some_bucket/

some_video.mp4 -> s3://some_bucket/some_video.mp4  [part 1 of 52, 5MB]
 5242880 of 5242880   100% in  164s    31.08 kB/s  done
some_video.mp4 -> s3://some_bucket/some_video.mp4  [part 2 of 52, 5MB]
 5242880 of 5242880   100% in  193s    26.46 kB/s  done
some_video.mp4 -> s3://some_bucket/some_video.mp4  [part 3 of 52, 5MB]
 2023424 of 5242880    38% in  135s    14.59 kB/s^CERROR: 
some_video.mp4' part 3 failed. Use
  /usr/local/bin/s3cmd abortmp s3://some_bucket/some_video.mp4 XXX_SOME_HASH_XXX
to abort the upload, or
  /usr/local/bin/s3cmd --upload-id XXX_SOME_HASH_XXX put ...
to continue the upload.
See ya!

Then I resume.

/usr/local/bin/s3cmd --upload-id XXX_SOME_HASH_XXX put --multipart-chunk-size-mb=5 some_video.mp4 s3://some_bucket/

Answer 2

I believe in Cyberduck in the transfers window, you can right click and select resume.

If that doesn't work, Cloudberry suports resuming uploads

Answer 3

You can use FileZilla Pro to transfer files to and from a S3 bucket. FileZilla Pro supports multipart upload and in case of failure it will resume the transfer.

For the records it comes with a lot of other features: large file support, bulk transfers, filters, directory comparison, remote file search, drag&drop, speed limits configuration.

I'm a member of FileZilla Pro team.

Learn more at https://filezillapro.com and at https://youtube.com/c/FileZillaPro