16

In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?

For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz

How do I do that without turning it into a tar.gz locally? (I want to save local space, as I don't have much space on my HDD.)

2 Answers 2

25

What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk.

tar c /var/test | gzip | aws s3 cp - "s3://tests/test1.tar.gz"

Breaking this down (where stdin and stdout refer to the standard input/output streams via the pipeline):

  • tar c /var/test creates a tar archive out of /var/test and outputs it to stdout...
  • ...which is read by gzip from stdin, and the gzipped file (.tar.gz) is output to stdout...
  • ...which is read by aws s3 cp - "s3://tests/test1.tar.gz" from stdin and sent to S3. The - tells the AWS CLI to copy from stdin.

This still performs the gzip operation locally, but does not require the creation of a temporary file, since the entire stream is sent straight over the network.

14
  • Bob, this answer looks like it's correct for SSHing files to other servers, but doesn't seem to address the question of how to upload to S3. It's probably a reasonably simple extension for someone who understands the S3 command line tools to apply this technique.
    – Tim
    Sep 8, 2017 at 0:59
  • @Tim ...somehow, I completely missed that. I'll update.
    – Bob
    Sep 8, 2017 at 1:04
  • 1
    @Tim Fixed. Probably only looked at the AWS bit and assumed EC2 while half asleep last night.
    – Bob
    Sep 8, 2017 at 1:11
  • 1
    A few questions about this solution: - will it work with directories too? - will the entire contents of the files be loaded in memory? Doesn't this give problems with large files? - is there any way to see progress?
    – murze
    Jun 10, 2018 at 9:51
  • 1
    @murze (1) of course, that's the whole point of packaging, (2) no, (3) no, (4) no.
    – Liz
    Sep 12, 2018 at 17:50
8
tar cvfz - /var/test | aws s3 cp - s3://tests/test1.tar.gz

You don't have to separately gzip; tar does that for you with the z option.

This works both in directions.

3
  • Is there a reverse command that can download the .tar.gz from S3 and decompress without having an intermediate file?
    – Pablote
    Jun 5, 2022 at 19:29
  • aws s3 cp s3://example-bucket/file.tar.gz - | tar -xz stackoverflow.com/a/46239830/5718301 Nov 1, 2022 at 23:55
  • Will this affect the number of API requests sent to S3 vs directly uploading the local .tar.gz file?
    – Marry Jane
    Sep 10, 2023 at 14:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .