Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've got several large files sitting in my linux hosted account that I need to upload to my S3 account. I dont want to download them first and then upload it into S3.

Is there any way I can "upload" it via the linux command line environment? Or access it via a website working with lynx?

share|improve this question
up vote 21 down vote accepted

http://s3tools.org/s3cmd does what you want to. Uploading & downloading files, syncing directories and creating buckets.

share|improve this answer
    
works like a charm! – siliconpi May 6 '11 at 8:42

Amazon provides their own CLI tools now too.

From http://aws.amazon.com/cli/

Using familiar syntax, you can view the contents of your S3 buckets in a directory-based listing.

$ aws s3 ls s3://mybucket
      LastWriteTime     Length Name
      -------------     ------ ----
                           PRE myfolder/
2013-09-03 10:00:00       1234 myfile.txt
...

You can perform recursive uploads and downloads of multiple files in a single folder-level command. The AWS CLI will run these transfers in parallel for increased performance.

$ aws s3 cp myfolder s3://mybucket/myfolder --recursive
upload: myfolder/file1.txt to s3://mybucket/myfolder/file1.txt
upload: myfolder/subfolder/file1.txt to s3://mybucket/myfolder/subfolder/file1.txt
...

A sync command makes it easy to synchronize the contents of a local folder with a copy in an S3 bucket.

$ aws s3 sync myfolder s3://mybucket/myfolder --exclude *.tmp
upload: myfolder/newfile.txt to s3://mybucket/myfolder/newfile.txt
...

Documentation for file related commands is here.

share|improve this answer

If you can't (perhaps you're on a shared host) or don't want to install extra tools, it is possible to just use bash, curl, and openssl.

http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash

file=/path/to/file/to/upload.tar.gz
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -L -X PUT -T "${file}" \
  -H "Host: ${bucket}.s3.amazonaws.com" \
  -H "Date: ${dateValue}" \
  -H "Content-Type: ${contentType}" \
  -H "Authorization: AWS ${s3Key}:${signature}" \
  https://${bucket}.s3.amazonaws.com/${file}

Note that I modified this script from the one in the above link. I added the -L option because AWS may insert a redirect in there. The -L option will follow the redirect for you.

One other caveat. This won't work for files larger than 5GB. Those require a multi-part upload that would require a more complex script.

share|improve this answer

A POSIX-compliant shell script that requires openssl, curl and sed only; supporting AWS Signature Version 4, which is required for region eu-central-1 (Frankfurt) and recommended for the others:

https://gist.github.com/vszakats/2917d28a951844ab80b1

#!/bin/sh

# To the extent possible under law, Viktor Szakats (vszakats.net) 
# has waived all copyright and related or neighboring rights to this 
# script.
# CC0 - https://creativecommons.org/publicdomain/zero/1.0/

# Upload a file to Amazon AWS S3 using Signature Version 4
#
# docs:
#    https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
#
# requires:
#    curl, openssl, sed, LF EOLs in this file

fileLocal="${1:-example-local-file.ext}"
bucket="${2:-example-bucket}"
region="${3}"
storageClass="${4:-STANDARD}"  # or 'REDUCED_REDUNDANCY'

awsStringSign4() {
   local kSecret kDate kRegion kService kSigning signedString
   kSecret="AWS4$1"
   kDate=$(printf         '%s' "$2" | openssl dgst -hex -sha256 -mac HMAC -macopt "key:${kSecret}"     2>/dev/null | sed 's/^.* //')
   kRegion=$(printf       '%s' "$3" | openssl dgst -hex -sha256 -mac HMAC -macopt "hexkey:${kDate}"    2>/dev/null | sed 's/^.* //')
   kService=$(printf      '%s' "$4" | openssl dgst -hex -sha256 -mac HMAC -macopt "hexkey:${kRegion}"  2>/dev/null | sed 's/^.* //')
   kSigning=$(printf 'aws4_request' | openssl dgst -hex -sha256 -mac HMAC -macopt "hexkey:${kService}" 2>/dev/null | sed 's/^.* //')
   signedString=$(printf  '%s' "$5" | openssl dgst -hex -sha256 -mac HMAC -macopt "hexkey:${kSigning}" 2>/dev/null | sed 's/^.* //')
   printf '%s' "${signedString}"
}

iniGet() {
   # based on: https://stackoverflow.com/questions/22550265/read-certain-key-from-certain-section-of-ini-file-sed-awk#comment34321563_22550640
   printf '%s' "$(sed -nr "/\[$2\]/,/\[.*\]/{/$3/s/(.*)=[ \\t]*(.*)/\2/p}" "$1")"
}

# Initialize access keys

if [ -z "${AWS_CONFIG_FILE}" ] ; then
   if [ -z "${AWS_ACCESS_KEY}" ] ; then
      echo 'AWS_CONFIG_FILE or AWS_ACCESS_KEY/AWS_SECRET_KEY envvars not set.'
      exit 1
   else
      awsAccess="${AWS_ACCESS_KEY}"
      awsSecret="${AWS_SECRET_KEY}"
      awsRegion='us-east-1'
   fi
else
   awsProfile='default'

   # Read standard aws-cli configuration file
   # pointed to by the envvar AWS_CONFIG_FILE
   awsAccess="$(iniGet "${AWS_CONFIG_FILE}" "${awsProfile}" 'aws_access_key_id')"
   awsSecret="$(iniGet "${AWS_CONFIG_FILE}" "${awsProfile}" 'aws_secret_access_key')"
   awsRegion="$(iniGet "${AWS_CONFIG_FILE}" "${awsProfile}" 'region')"
fi

# Initialize defaults

fileRemote="${fileLocal}"

if [ -z "${region}" ] ; then
   region="${awsRegion}"
fi

echo "Uploading" "${fileLocal}" "->" "${bucket}" "${region}" "${storageClass}"

# Initialize helper variables

httpReq='PUT'
authType='AWS4-HMAC-SHA256'
service='s3'
baseUrl=".${service}.amazonaws.com"
dateValueS=$(date -u +'%Y%m%d')
dateValueL=$(date -u +'%Y%m%dT%H%M%SZ')
if hash file 2>/dev/null ; then
   contentType="$(file -b --mime-type "${fileLocal}")"
else
   contentType='application/octet-stream'
fi

# 0. Hash the file to be uploaded

if [ -f "${fileLocal}" ] ; then
   payloadHash=$(openssl sha256 -hex < "${fileLocal}" 2>/dev/null | sed 's/^.* //')
else
   echo "File not found: '${fileLocal}'"
   exit 1
fi

# 1. Create canonical request

# NOTE: order significant in ${headerList} and ${canonicalRequest}

headerList='content-type;host;x-amz-date;x-amz-server-side-encryption;x-amz-storage-class'

canonicalRequest="\
${httpReq}
/${fileRemote}

content-type:${contentType}
host:${bucket}${baseUrl}
x-amz-date:${dateValueL}
x-amz-server-side-encryption:AES256
x-amz-storage-class:${storageClass}

${headerList}
${payloadHash}"

# Hash it

canonicalRequestHash=$(printf '%s' "${canonicalRequest}" | openssl sha256 -hex 2>/dev/null | sed 's/^.* //')

# 2. Create string to sign

stringToSign="\
${authType}
${dateValueL}
${dateValueS}/${region}/${service}/aws4_request
${canonicalRequestHash}"

# 3. Sign the string

signature=$(awsStringSign4 "${awsSecret}" "${dateValueS}" "${region}" "${service}" "${stringToSign}")

# Upload

curl -s -L -X "${httpReq}" -T "${fileLocal}" \
   -H "Content-Type: ${contentType}" \
   -H "Host: ${bucket}${baseUrl}" \
   -H "X-Amz-Date: ${dateValueL}" \
   -H "X-Amz-Storage-Class: ${storageClass}" \
   -H "X-Amz-Server-Side-Encryption: AES256" \
   -H "X-Amz-Content-SHA256: ${payloadHash}" \
   -H "Authorization: ${authType} Credential=${awsAccess}/${dateValueS}/${region}/${service}/aws4_request, SignedHeaders=${headerList}, Signature=${signature}" \
   "https://${bucket}${baseUrl}/${fileRemote}"

Notice, the script will enable server-side AES256 encryption by default.

share|improve this answer
    
For the search engines: This is the correct solution for eu-central-1 and in general if you get the error The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256 – Steen Feb 17 at 12:08

Alternatively you can try https://github.com/minio/mc

mc provides minimal tools to work with Amazon S3 compatible cloud storage and filesystems. It has features like resumable uploads, progress bar, parallel copy. mc is written in Golang and released under Apache license v2.

share|improve this answer

I've found Python's AWS bindings in the boto package (pip install boto) to be helpful for uploading data to S3.

The following script can be called like: python script_name.py "sub_bucket_name" "*.zip" where sub_bucket_name indicates the name of the directory in which the files should be stored in S3, and *.zip is a glob path designating one or more files to be uploaded:

import sys, glob, os, boto
from boto.s3.key import Key

def percent_cb(complete, total):
    sys.stdout.write('.')
    sys.stdout.flush()

id = '< your id here >'               # AWS Access Key ID
secret = '< your secret here >'       # AWS Secret Access Key
bucket_name = '< your bucket here >'  # Bucket wherein content will be stored
conn = boto.connect_s3(id, secret)    # Establish a connection to S3
bucket = conn.get_bucket(bucket_name, validate=False)  # Connect to bucket
k  = Key(bucket)                      # Connect to the bucket's key

for i in glob.glob(sys.argv[2]):      # Read in files to push to S3

        sub_bucket = sys.argv[1]  # Directory within bucket where files will be stored
        k.key = sub_bucket + "/" + os.path.basename(i) # Path each uploaded file will have on S3

        k.set_contents_from_filename(i, cb=percent_cb, num_cb=10)  # Push data to S3

        print 'Uploading %s to Amazon S3 bucket %s' % (i, bucket_name)  # Report status
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .