1

Basically I need to control how fast I upload files to AWS S3 - I have a folder with tens of thousands of files, and this is what I do now:

for f in *; do aws s3 cp $f s3://my-bucket; done;

What I'd like to do is like this pseudo code:

for f in *; 
   cnt=0
   do aws s3 cp $f s3://my-bucket
   if cnt % 1000 == 0
       sleep 5 min
done; 

Sorry for mixing python and shell syntax together but I hope I expressed my idea clearly. Please kindly help. Thanks!

2

This might do what you want.

#!/usr/bin/env bash

counter=1

for f in *; do
  echo aws s3 cp "$f" s3://my-bucket
  ((counter++ % 10 == 0)) && echo && sleep 5
done
  • That's just a dry-run of what should/could be executed.

  • Remove the echo in front of aws and the && echo if you think the output is ok.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.