Upload multiple files to S3 from stdin

Question

I found the option to upload to S3 from stdin:

cat test.txt | aws s3 cp - s3://mybucket/test.text

But I have a tgz archive coming in via curl and containing multiple files. I want to write the contents into an S3 bucket. This is what I would like to do:

curl http://rapidsai-data.s3-website.us-east-2.amazonaws.com/notebook-mortgage-data/mortgage_2000.tgz | tar xvz - | aws s3 sync - s3://mybucket/

However sync does not support stream input and cp does not support multiple files. Any recommendation?

Answer 1

I dont think the CLI supports multiple files from stdin by design.

Sync explicitly deals with one local directory and one S3 bucket - no mention of stdin support in docs. (https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html)

Copy works with a single local file and single remote S3 key and as you describe will accept input from stdin. (https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html)

As i see it you can either extract the files locally and use s3 sync (or extract the files and run a script to iterate each file and use s3 cp on each file).

If your tight on space you might consider uploading the TGZ via stdin to some temporary location in S3 then using a separate process to download, extract and upload the individual files - but theres no getting round the need to extract the files then upload AFAIK.