Skip to main content
25 votes

AWS upload folder to S3 as tar.gz without compressing locally

What you're really looking for is not saving a local file. You can use pipes to send the data from tar through gzip to s3 without saving anything to disk. tar c /var/test | gzip | aws s3 cp - "s3://...
Bob's user avatar
  • 61.9k
18 votes

Time machine backup on S3

Using AWS S3 as the storage of Mac Time Machine definitely works now. Time Machine can backup the data to external device, so the key is mounting S3 bucket as a POSIX FS in Mac. There are many tools ...
Kane's user avatar
  • 505
10 votes

How to automate regular Google Takeout backups to cloud storage

This is a partial answer with partial automation. It may stop working in the future if Google chooses to crack down on automated access to Google Takeout. Features currently supported in this answer:...
8 votes

AWS upload folder to S3 as tar.gz without compressing locally

tar cvfz - /var/test | aws s3 cp - s3://tests/test1.tar.gz You don't have to separately gzip; tar does that for you with the z option. This works both in directions.
Robv's user avatar
  • 81
8 votes

Can I use rsync for backing up at Amazon S3

Amazon's official command line interface worked well for me. It can sync from local to an S3 bucket, and also sync from an S3 bucket back to local files. 1) install Amazon's command line interface, ...
moltenform's user avatar
7 votes

Know the identity behind an AWS canonical ID

The AWS documentation at https://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example4.html says: The canonical user ID is the Amazon S3–only concept. It is a 64-...
Alastair Irvine's user avatar
5 votes

How to search for nested files in my S3 bucket?

There isn't a find equivalent that I know of. I do this: aws s3 ls s3://bucket/ --recursive | grep filename
user2711286's user avatar
4 votes
Accepted

curl can not download file but browser can

Use single quotes to make sure that any potentially special characters are taken literally. [user@localhost ~]# curl 'https://xxxxxxxxxx.s3.amazonaws.com/xxxx-xxxx-xxxx-xxxx/xxxxxxxxxxxxx/x?...
apocalysque's user avatar
  • 1,278
3 votes
Accepted

Determining the service that is using *.CloudFront.net

The IP addresses associated with *.cloudfront.net hostnames do indeed have the format you described: iii-ppp-aaa-ddd.locNN.r.cloudfront.net For example: server-13-32-14-48.muc51.r.cloudfront.net ...
Michael - sqlbot's user avatar
3 votes

multiple s3 users in one bucket or per-user buckers

You definitely do not need a bucket for each user. Never mind the fact that it seems very unlikely that AWS support would approve a request to increase your account's default total bucket limit from ...
Michael - sqlbot's user avatar
3 votes

Time machine backup on S3

Edit: I tried this and it didn't work. (Time Machine cannot see the mounted volume/bucket.) You may be able to use Panic's Transmit app to [mount an S3 bucket as a local Volume][1] and then point ...
Jay's user avatar
  • 145
3 votes

How to search for nested files in my S3 bucket?

From command line you can use command like: aws s3 ls s3://bucket/ --recursive "filename"
Romeo Ninov's user avatar
  • 6,415
2 votes

How to upload big files to S3 on a flaky connection?

You can use FileZilla Pro to transfer files to and from a S3 bucket. FileZilla Pro supports multipart upload and in case of failure it will resume the transfer. For the records it comes with a lot of ...
josuegomes's user avatar
2 votes

Can I use rsync for backing up at Amazon S3

I'm surprised nobody has mentioned rclone. It is like rsync but it connects to just about any cloud storage provider out there, and also supports full encryption. I use it regularly, it is well ...
gregtzar's user avatar
  • 121
2 votes
Accepted

Can I make an Amazon AWS Lamda function which is triggered by the creation of an S3 bucket?

EDIT As @michael-sqlbot has pointed out in the comments, what you are trying to achieve may be possible but with a multi-step process. CloudTrail can log S3 bucket-level events (https://docs.aws....
roryrjb's user avatar
  • 136
2 votes

copy specific files from s3 bucket

I'm not sure if awscli has a built-in way to do this, but I've always just used simple bash to do things like this. For example: for i in $(aws s3 ls s3://bucketname/ | grep 1400x1400); do aws s3 cp ...
Michael Pobega's user avatar
2 votes

Why CNAME is not working as supposed?

A CNAME is not a redirect, it's an alias. What it does is tell a DNS resolver that your s3.example.com address should use the same DNS information as amazon's s3 domain. This all happens at the DNS ...
kicken's user avatar
  • 1,753
2 votes

Why CNAME is not working as supposed?

The CNAME in your DNS zone is working as you expect, but browsers check that the certificate presented by the remote web site is valid for the domain being visited. The amazon s3 server where your ...
cas's user avatar
  • 671
2 votes
Accepted

Nginx SSL Preread Sporadically Gets Requests Where No Server Name Is Extracted

I am not familiar with Amazon S3 as such, but I think that the problem is that the requests are generated without setting the server name (see Server Name Indication). I experienced the very same ...
nehtor.t's user avatar
  • 166
2 votes

Get all files (or file names) out of s3 bucket for specific date

If the filename contains the date, you can use include and exclude filters: aws s3 cp s3:{path}/ {directoryToCopyTo} --exclude "*" --include "*2019-09-09*" If the date is only in ...
enharmonic's user avatar
2 votes

Is there a way we can way to minimize or eliminate warning when a recipient receives an email from AmazonSES

With quick googling it appears setting up SPF and DKIM should resolve it. When you first set up Amazon SES to send emails, you may find there is a "Via amazonses.com" warning for the email ...
Peregrino69's user avatar
  • 4,744
1 vote

S3FS ownership and permissions

Just for others coming to this... make sure you have "s3:PutObjectAcl" set on your policy for the relative role your creds are for.
Carl L's user avatar
  • 11
1 vote

AWS ACCESS KEY ISSUE

You needed to invalidate that key anyway, because you don't know who had access to it. Now you have to create a new access key on IAM and use it in your scripts. Make sure the new created key is ...
Algeriassic's user avatar
  • 1,659
1 vote

How to fix awscli s3 error "The Content-MD5 you specified did not match what we received" (BadDigest)?

This worked in part: md5 is wrong is should use sha256, setting this helps: aws configure set default payload_signing_enabled = true Second Solution Issue more in depth is related to: http://tracker....
user51070's user avatar
  • 135
1 vote

How to encypt files in transit to S3 from local macine

"By default, the AWS CLI uses SSL when communicating with AWS services. For each SSL connection, the AWS CLI will verify SSL certificates.." Source - AWS CLI documentation. Note that this SSL ...
Tim's user avatar
  • 614
1 vote

Can I host a static website on Amazon EC2 coupled with EFS?

Important note: As of 2018-07-12, EFS allows you to purchase provisioned throughput. The answer below reflects the behavior of the service before this feature was available. Previously, small EFS ...
Michael - sqlbot's user avatar
1 vote

Unable to import my vmdk to EC2

The error message suggests you are not in the correct region. According to the AWS documentation you need to specify a zone. You do this with the option flag -z zonename. In your instance I think it ...
Burgi's user avatar
  • 6,601

Only top scored, non community-wiki answers of a minimum length are eligible