12

In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?

For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz

How do I do that without turning it into a tar.gz locally? (I want to save local space, as I don't have much space on my HDD.)

19

What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk.

tar c /var/test | gzip | aws s3 cp - "s3://tests/test1.tar.gz"

Breaking this down (where stdin and stdout refer to the standard input/output streams via the pipeline):

  • tar c /var/test creates a tar archive out of /var/test and outputs it to stdout...
  • ...which is read by gzip from stdin, and the gzipped file (.tar.gz) is output to stdout...
  • ...which is read by aws s3 cp - "s3://tests/test1.tar.gz" from stdin and sent to S3. The - tells the AWS CLI to copy from stdin.

This still performs the gzip operation locally, but does not require the creation of a temporary file, since the entire stream is sent straight over the network.

9
  • Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique. – Tim Sep 8 '17 at 0:59
  • @Tim ...somehow, I completely missed that. I'll update. – Bob Sep 8 '17 at 1:04
  • 1
    @Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night. – Bob Sep 8 '17 at 1:11
  • 1
    A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress? – murze Jun 10 '18 at 9:51
  • 1
    @murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no. – Ekevoo Sep 12 '18 at 17:50
4

tar cvfz - /var/test | aws s3 cp - s3://tests/test1.tar.gz

you dont have to separately gzip, tar does that for you with the z toggle...

This works both directions, I use it almost daily..

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.