Questions tagged [amazon-s3]
Amazon S3 (Simple Storage Service) is a web service which provides storage and various interfaces for accessing data stored there.
176
questions
0
votes
1
answer
763
views
How to auto upload a file to s3
How can I upload a json file to my s3 in easiest way?
Basically there is a json file in a webpage and I want to keep copy it every 5 minutes and upload to my s3.
- 3
0
votes
1
answer
1k
views
Linux - How to disable disk cache for a specific mount point?
Is it possible to disable disk cache for a mount point on Linux system?
I have a s3fs mount on multiple systems and i need to disable disk cache for it. how can i achieve this?
0
votes
0
answers
690
views
goofys mount point on multiple linux systems
I have a Minio bucket which is mounted on three different linux systems. the problem is when a file changes on one of mount points but file size doesn't changes that change doesn't get reflected on ...
2
votes
1
answer
7k
views
How to fix awscli s3 error "The Content-MD5 you specified did not match what we received" (BadDigest)?
Uploading a file over 10 GB to s3 fails with:
aws s3 cp 'foo.log.gz' s3://mybucket/2018/12/ --storage-class REDUCED_REDUNDANCY
upload failed: ./foo.log.gz to s3://mybucket/2018/12/foo.log.gz
An ...
- 135
2
votes
2
answers
6k
views
copy specific files from s3 bucket
I have some files in s3 bucket.
aws s3 ls s3://bucketname/
file-111-100x100.jpg
file-112-1400x1400.jpg
file-123-250x250.jpg
file-231-1400x1400.jpg
file-222-700x700.jpg
file-333-100x100.jpg
file-131-...
- 63
2
votes
2
answers
810
views
Why CNAME is not working as supposed?
I have set a CNAME entry to redirect over Amazon S3 bucket, however there is something that doesnt works as supposed (by me of course) with CNAME I expect to redirect a fictional subdomain, say for ...
- 83
2
votes
2
answers
16k
views
curl can not download file but browser can
URL example: https://xxxxxxxxxx.s3.amazonaws.com/xxxx-xxxx-xxxx-xxxx/xxxxxxxxxxxxx/x?AWSAccessKeyId=AKIAIFKEBR4CLVTDYSLA&Signature=Kv%2B859dJ2eUmLgHdkT%2F9nUJEeRM%3D&Expires=1922517562
I ...
- 190
2
votes
1
answer
464
views
Jungledisk does not restore my files. How can this be fixed?
I backup my files with Jungledisk for many years now, my backup is on Amazon S3 and is ~600 GB in size. Unluckily there came the time when I need to restore some files (my email files got damaged).
...
- 121
0
votes
0
answers
470
views
Amazon CloudFront buffering while streaming video to Roku
I'm looking for help on trying to troubleshoot a buffering issue using Amazon's CloudFront service. I've set up an S3 bucket and have it associated with a CloudFront service. I use the service to ...
- 672
2
votes
1
answer
6k
views
Determining the service that is using *.CloudFront.net
How can I determine that accessing to cloudfront.net is associated to which service, software or owner?
(as far as I understand and the normal who-is lookup tells, it is related to Amazons' cloud ...
- 182
16
votes
2
answers
19k
views
AWS upload folder to S3 as tar.gz without compressing locally
In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?
For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz
How do I do that ...
- 161
1
vote
0
answers
117
views
Meteor(Blaze) : CFS: S3 Create thumbnail Image from Video and upload both to S3
I am using CFS packages to upload videos to Amazon S3, but it takes a long time to load the videos on Amazon when they are of 10-20 MB. Till then, I want to use the videos' first frame and when ...
- 111
0
votes
2
answers
1k
views
How to encypt files in transit to S3 from local macine
I have installed AWS CLI in a server in my on-prem server I am able to copy files via the cli S3 commands, how can i know if the files in transit to S3 are encrypted?. There is a SSL installed on the ...
- 1
0
votes
0
answers
301
views
Large files split when uploaded to cloud (B2 Backblaze)
I'm uploading some ISO files which I made from DVDs to B2 Backblaze cloud. But I see they split there:
There are also some 0 bytes size files, which seems strange.
How can this be explained?
I'm ...
- 604
0
votes
1
answer
147
views
AWS S3 bucket size reduced after configuring life cycle with Transisition to Standard IA class
Few days back I tried to configure a life cycle for S3 bucket intending to move old files to Glacier.
I configured a LifeCycle for that bucket and added transistion to Standard IA class after 31 days ...
-1
votes
1
answer
450
views
Can I make an Amazon AWS Lamda function which is triggered by the creation of an S3 bucket? [closed]
I would like to send a mail when a new bucket is created on our AWS account. I'd prefer this to be near-instantaneous so preferably not time scheduled, but triggered by the CreatBucket event directly.
...
- 753
0
votes
1
answer
69
views
Configure root login bucket for an s3 user?
I want to configure a policy/permission for a specific s3 user so they will login to a particular sub folder under a bucket.
I've got a bucket location here: s3://foo.s3.amazonaws.com and underneath ...
- 311
-2
votes
3
answers
2k
views
How to implement anti-scraping mechanisms for my Amazon S3 based site?
I have a few static web pages hosted on amazon S3 that are very frequently updated. I want to implement few anti-scraping mechanisms like banning IP's which are making too many requests or making ...
- 51
6
votes
0
answers
2k
views
AWS Client S3 mv command is slow
I am using aws cli to move the files around in S3 in the same bucket. As far as I know, AWS S3 does not maintain the folder structure. It stores the entire path as key to the file and displays it in ...
- 123
4
votes
1
answer
5k
views
multiple s3 users in one bucket or per-user buckers
intelligent people, noob here!
I am planning on building a multi-user apparatus on s3 for photo/object storage, and I was planning on using s3. I have the whole front-end planned out but I have a ...
- 141
1
vote
1
answer
730
views
Can I host a static website on Amazon EC2 coupled with EFS?
My website is currently hosted on EC2 with EBS volume. But very often I run a java program on the same instance (Which outputs to EBS) to generate/update files for the website. CPU & memory peaks ...
- 51
2
votes
3
answers
2k
views
S4cmd working only for Amazon S3. Is it possible to configure it for DreamObjects cloud storage the way S3cmd worked?
I just switched from S3cmd to S4cmd. It's working great for EC2 to Amazon S3 data transfer. But when I'm trying DreamObjects or CenturyLink Object storage in it's place, it shows the below error:
...
- 51
0
votes
2
answers
152
views
Centralizing data AWS
I want to centralizer about 1TB of data in AWS for 3 on premise locations (5 users each location). We are going growing but still small. Each location has a 30/5mpbs connection. Each has its own Linux ...
- 644
0
votes
0
answers
3k
views
No permission to download my own file in AWS S3
I can see I have the all the checkboxes checked in S3 panel with me as the Grantee.
But when I try access the URL or downloading the file, it says access is denied.
I don't want to change permission ...
- 931
1
vote
1
answer
430
views
Unable to import my vmdk to EC2 [duplicate]
I am trying to import my vmdk to EC2 as a instance using S3 bucket
I am using this syntax
ec2-user@ip-10-0-x-x ~$ ec2-import-instance disk1.vmdk -f VMDK -t m1.xlarge -a x86_64 -b import-raj -o ...
- 11
0
votes
0
answers
92
views
keep amazon s3 bucket public between uploads
I've setup an Amazon CloudFront CDN to point to an Amazon S3 Bucket. It works great, I love how it's streamlined much of my process!
But I have a problem, every time I do a new upload to it, the new ...
- 233
1
vote
1
answer
2k
views
How can I restore a MySql database dump direct from S3?
I have a gzipped database backup stored on S3 and would like to restore it to MySql without having to first download it due to disk space constraints. I tried the two commands below but got gzip: ...
- 147
1
vote
0
answers
1k
views
does s3cmd put command overwrite files?
I am trying to write a script that will automatically backup a drive I have to amazon s3. I have downloaded s3cmd and I am wondering what the put command does...
It seems to me like it uploads the ...
- 227
0
votes
1
answer
1k
views
Would be great to send a large file to AWS EC2
Background:
I have battled to install an rstudio server on AWS EC2 for some time (since November). It has been like the peeling back layers of an onion, always another hurdle. For a windows user ...
- 1
1
vote
1
answer
4k
views
s3 file access denied for large files using cyberduck
When I use a custom policy below, I get the following error when uploading large files (1G and over) using cyberduck
file access denied. please contact your web hosting service provider
for ...
- 141
1
vote
1
answer
453
views
How to copy a .zip from within a .tar to a given location without extracting entire .tar using AWS S3 CLI
OSX 10.11 - python3.5 or AWS CLI (or other tool?)
I have ~ 5,000 subdirectories within an Amazon S3 bucket, each subdirectory contains a single .tar. In each .tar it contains only one .zip, ~<1mb ...
- 113
0
votes
1
answer
339
views
How to safely upload many GB of data to Amazon S3?
I am backing up my personal photo/video library to Amazon S3 (several hundred GB).
I am currently using Cyberduck on my Mac to drag & drop, but I've already had a few fail without a clear retry/...
- 14.9k
9
votes
5
answers
3k
views
Let's Encrypt with Amazon S3
Will it be possible to have https on the my static Amazon S3 website with Let's Encrypt?
Even I have Let's Encrypt Closed Beta Invite now. But how to use it with Amazon S3?
- 111
1
vote
0
answers
142
views
AWS server retired; new instance lost access to S3
My old company has a system with a node.js front end running on an AWS EC2 instance. The instance was 'retired' yesterday morning, and I got a frantic call from the CEO because they were down.
I got ...
- 431
1
vote
0
answers
564
views
How to transcode AES encrypted files with AWS Elastic Transcoder
I want to upload my encrypted videos to S3 and use Elastic Transcoder to transcode them. This is what I'm currently doing and isn't working at all. I'm using openssl to encrypt my video files. The ...
- 113
0
votes
1
answer
168
views
Geo-balanced (static) website hosting
Abstract
I'm rebuilding my website and this time around, I have decided to go with a static website generator (jekyll, sculpin.io, etc). Partially as an excercise, I want this site to be hosted on a ...
- 817
1
vote
3
answers
828
views
Renaming files fetched via “wget --mirror” in Bash before uploading to an Amazon S3 statically hosted area
I’m trying to archive and upload an old website to statically hosted Amazon S3 area.
I was able to get the contents with wget, using the following command:
wget --mirror --no-parent --html-extension ...
- 892
0
votes
1
answer
537
views
S3Fox Exception: Too Many Arguments
I'm running the S3Fox add-on in FireFox 38.0.1
Getting this error when I try to download a file from Amazon S3 to my local PC:
[Exception... "Not enough arguments [nsIWebBrowserPersist.saveURI]" ...
- 403
1
vote
2
answers
3k
views
Accsess S3 from the command line without a secret key
I have some files on a S3 account that I don’t own and therefore do not have the secret key for. Using a link that was sent to me, I can enter the “Account Number”, user name and password I can login ...
- 142
1
vote
1
answer
4k
views
How can I import a VMDK disk image to an AWS EC2 t2.micro instance on us-east-1e region using an S3 bucket on us-east-1?
I've recently purchased a reserved EC2 micro instance on region eu-east-1e.
I am having trouble importing a virtual machine using ec2-api-tools:
ec2iin -t t2.micro "CentOS 7 x86_64-disk1.vmdk" -g my-...
- 577
1
vote
1
answer
3k
views
How to download a large S3 object from the command line much faster than s3cmd
With the s3cmd command you can download files from S3 like this:
s3cmd get s3://my.private.bucket/some/location/hugefile.dat
But that can take quite a while if the stored files are really big.
Are ...
- 385
1
vote
1
answer
1k
views
File permissions with s3fs and ubuntu
I followed this guide to set up an amazon s3 upload procedure for my ember-js application with grunt-s3:
http://www.octolabs.com/blogs/octoblog/2014/05/24/deploying-ember-cli-to-amazon-s3-with-grunt/
...
7
votes
0
answers
17k
views
Can't connect to S3 using cyberduck
I'm trying to connect to my S3 bucket using cyberduck.
I select S3 from the dropdown.
Input into server name: {bucket-name}.s3.amazonaws.com
Access key ID and Secret Access Key into the next two ...
- 171
3
votes
1
answer
2k
views
AWS S3 cp does not download 40 GB file
I just uploaded a 40GB backup file from my linux EC2 machine to an S3 bucket. However when I want to download it the aws command ends with no message and nothing was copied.
Here is my command:
...
- 193
3
votes
1
answer
2k
views
https proxy s3 aws via nginx with pagespeed
I'm trying to setup pagespeed to proxy s3 files via nginx. I want to "save" the files on amazon s3;
The story I want is:
User A
1. nginx serves page to user; and rewrites all https://s3.amazonaws....
- 163
1
vote
1
answer
2k
views
How to get the last time a particular file was downloaded from S3
I was wondering if there is a way to get the last downloaded time on of an S3 object.
I had a use case where I need this data to decide whether to remove a file if its not downloaded for a set period.
- 13
1
vote
1
answer
4k
views
Sync local files to S3 similar to Robocopy
I am looking for a way to synchronize an entire local folder structure to Amazon S3, similar to how one might synchronize two folders using Robocopy.
Whatever solution I come up with needs to be ...
- 225
0
votes
0
answers
668
views
Tar a very large file directly to a drive
I was running my code to download some data on the Amazon cloud. The instance that I was running had an 8GB storage that I had bought with it plus 140GB extra instance storage (sort of plugged in as a ...
- 31
1
vote
0
answers
3k
views
How to use FFMPEG to transcode mp4 video for http stream in Amazon S3?
I want to use FFMPEG to transcode my mp4 video for http streaming on Amazon S3 without downloading. However, with reference to this post, I still cannot get it streaming:
ffmpeg -i input.avi -c:v ...
- 55
1
vote
1
answer
1k
views
Arq app — Merge a backup folder from a fresh install
context
I'm using Arq app, my disk have 2 partitions, os and data, the partition I'm backing up is data.
added the content under data partition to backup and waited for the data upload to complete
...
- 168