0

I found the option to upload to S3 from stdin:

cat test.txt | aws s3 cp - s3://mybucket/test.text

But I have a tgz archive coming in via curl and containing multiple files. I want to write the contents into an S3 bucket. This is what I would like to do:

curl http://rapidsai-data.s3-website.us-east-2.amazonaws.com/notebook-mortgage-data/mortgage_2000.tgz | tar xvz - | aws s3 sync - s3://mybucket/

However sync does not support stream input and cp does not support multiple files. Any recommendation?

0

I dont think the CLI supports multiple files from stdin by design.

Sync explicitly deals with one local directory and one S3 bucket - no mention of stdin support in docs. (https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html)

Copy works with a single local file and single remote S3 key and as you describe will accept input from stdin. (https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html)

As i see it you can either extract the files locally and use s3 sync (or extract the files and run a script to iterate each file and use s3 cp on each file).

If your tight on space you might consider uploading the TGZ via stdin to some temporary location in S3 then using a separate process to download, extract and upload the individual files - but theres no getting round the need to extract the files then upload AFAIK.

| improve this answer | |
  • Thx... but I wouldn't bother if that were an option. The data to be transferred is a bit too big for that. I think so who created the data source should have thought about that. – Marcel Boldt May 27 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.