s3cmd sync hangs at "Retrieving list of remote files for ..."

Question

As part of a backup solution, I am using s3cmd to transfer a load of files.

I have four different jobs with directories of different sizes and files of different sizes.

Three of the jobs runs just fine. The last job, though, hangs at the message:

Retrieving list of remote files for <...>

And when I say it hangs, I mean that it doesn't go any further. It stood frozen like this for a full week on a 100% stable office internet connection.

Now, the directory it is trying to upload is a large one. About 306GB. This is by far the largest of the jobs.

I saw a post on StackOverflow with a problem similar (not identical) to this, and the accepted answer said to edit .s3cfg and set a bigger socket_timeout. I changed it from 10 to 180, but it has not made a difference.

Any idea what to try next? I've come up short in my googling.

Answer 1

The solution to this issue was, for me, to stop using s3cmd and start using the aws tool instead.

instead of s3cmd sync ..., I now use aws s3 sync .... This works like a charm. I wish I'd never stumbled upon s3cmd.

Answer 2

As the destination directory becomes large the time to retrieve the list of md5 and size data increases.

For me similarly large backups are completing this step in less than twenty minutes. I notice that I have the socket_timeout set to 300.

You can also avoid the md5 scan of each file in the target bucket by using --no-check-md5 but I have not found it necessary to do this.