Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've got several large files sitting in my linux hosted account that I need to upload to my S3 account. I dont want to download them first and then upload it into S3.

Is there any way I can "upload" it via the linux command line environment? Or access it via a website working with lynx?

share|improve this question

3 Answers 3

up vote 14 down vote accepted

http://s3tools.org/s3cmd does what you want to. Uploading & downloading files, syncing directories and creating buckets.

share|improve this answer
    
works like a charm! –  siliconpi May 6 '11 at 8:42

Amazon provides their own CLI tools now too.

From http://aws.amazon.com/cli/

Using familiar syntax, you can view the contents of your S3 buckets in a directory-based listing.

$ aws s3 ls s3://mybucket
      LastWriteTime     Length Name
      -------------     ------ ----
                           PRE myfolder/
2013-09-03 10:00:00       1234 myfile.txt
...

You can perform recursive uploads and downloads of multiple files in a single folder-level command. The AWS CLI will run these transfers in parallel for increased performance.

$ aws s3 cp myfolder s3://mybucket/myfolder --recursive
upload: myfolder/file1.txt to s3://mybucket/myfolder/file1.txt
upload: myfolder/subfolder/file1.txt to s3://mybucket/myfolder/subfolder/file1.txt
...

A sync command makes it easy to synchronize the contents of a local folder with a copy in an S3 bucket.

$ aws s3 sync myfolder s3://mybucket/myfolder --exclude *.tmp
upload: myfolder/newfile.txt to s3://mybucket/myfolder/newfile.txt
...

Documentation for file related commands is here.

share|improve this answer

If you can't (perhaps you're on a shared host) or don't want to install extra tools, it is possible to just use bash, curl, and openssl.

http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash

file=/path/to/file/to/upload.tar.gz
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -L -X PUT -T "${file}" \
  -H "Host: ${bucket}.s3.amazonaws.com" \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  https://${bucket}.s3.amazonaws.com/${file}

Note that I modified this script from the one in the above link. I added the -L option because AWS may insert a redirect in there. The -L option will follow the redirect for you.

One other caveat. This won't work for files larger than 5GB. Those require a multi-part upload that would require a more complex script.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.