Questions tagged [amazon-s3]
Amazon S3 (Simple Storage Service) is a web service which provides storage and various interfaces for accessing data stored there.
176
questions
0
votes
2
answers
1k
views
make new objects in s3 bucket public by default
I have an s3 bucket in which the objects within it need to be made public, right now anytime I upload a new file I have to click on it and then click "make public using acl" I wonder if ...
0
votes
0
answers
115
views
ffmpeg stream image flow to AWS S3
For a video monitoring solution, I want to push thumbnails of an incoming video (TS/UDP) to a S3 bucket from where those thumbnails are shown on a web page
Currently the following bash loop is working:...
0
votes
0
answers
110
views
AWS S3 <-> MS Onedrive sync with rules
Is it possible to have an automatic synchronization between a specific AWS S3 bucket and a OneDrive folder ? and to specify rules e.g. remove certain files extensions from the sync ; only sync ...
1
vote
1
answer
632
views
How does FFMpeg read from a remote s3 url
I'm intrigued how does FFMpeg read a video file from a remote url (for example, an aws s3 presigned url).
Does it download the file in chunks and then process it chunk by chunk?
Or does it download ...
0
votes
0
answers
24
views
Mount or sync an S3 versioned bucket with rewind - back in time
Assuming I have an S3 versioned bucket (backed by Ceph), I would like to restore from it at a specific time. mcli has a --rewind feature that allows some operations to time-travel into the past taking ...
0
votes
0
answers
53
views
How to route linode object bucket to domain name?
I have a linode object bucket up and running with my files/folders. I have purchased the domain and have set a CNAME to point to [subdomain].[domain].com.website-us-southeast-1.linodeobjects.com. When ...
85
votes
7
answers
156k
views
Uploading files to S3 account from Linux command line
I've got several large files sitting in my Linux hosted account that I need to upload to my S3 account. I don't want to download them first and then upload them into S3.
Is there any way I can "...
1
vote
1
answer
83
views
Connect to S3 Web Console with Access and Secret Key
Is it possible to authenticate in the Amazon S3 console (https://s3.console.aws.amazon.com) using an access key/secret key pair? It looks like it only accepts user credentials. If not, is there any ...
2
votes
1
answer
7k
views
How to fix awscli s3 error "The Content-MD5 you specified did not match what we received" (BadDigest)?
Uploading a file over 10 GB to s3 fails with:
aws s3 cp 'foo.log.gz' s3://mybucket/2018/12/ --storage-class REDUCED_REDUNDANCY
upload failed: ./foo.log.gz to s3://mybucket/2018/12/foo.log.gz
An ...
1
vote
0
answers
47
views
AWS Windows EC2 File Transfer - Need to keep session open
I have a rather large file that I need to copy from an S3 bucket to a Windows EC2 and it will not complete before a session ends. I have found lots of suggestions for this for Linux but not Windows.
...
33
votes
5
answers
56k
views
Amazon S3 Recover Deleted File
Is it possible to recover a file I deleted from my S3 bucket?
0
votes
1
answer
351
views
aws sync doesn't download the file and instead create an empty directory
The file I'm trying to download is:
s3://ont-open-data/gm24385_q20_2021.10/analysis/20210805_1713_5C_PAH79257_0e41e938/guppy_5.0.15_sup/align_unfiltered/chr19/calls2ref.bam
This is an open dataset. I ...
0
votes
1
answer
259
views
How translate s3 url to http in chrome automatically?
Let say I have a link like that:
s3://mybucket/mybucket?prefix=myprefix/file.json
And I open it in chrome browser. I will get an error. Is it possible to automatically translate the s3 url from
...
1
vote
0
answers
257
views
AWS S3 Bucket CORS Error
I have an amazon s3 bucket called cs-ia.
This is my CORS configuration:
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
...
16
votes
2
answers
19k
views
AWS upload folder to S3 as tar.gz without compressing locally
In AWS CLI, how do I upload a folder as a tar.gz file without creating a tar.gz locally?
For example, I have a folder at /var/test and I want to upload it to /tests/test1.tar.gz
How do I do that ...
6
votes
1
answer
12k
views
Why can't I connect to my Amazon S3 storage with two of my three clients?
I use Transmit, Cyberduck, and Espresso, all of which support connecting to Amazon S3. But for some reason, I can only connect to my S3 storage successfully in Transmit; Cyberduck and Espresso both ...
0
votes
2
answers
9k
views
s3cmd: how to copy files from local to S3 bucket with existing structure
I'm trying to copy files with s3cmd to a bucket with an existing folder structure. Both structures already exist both on local dev machine and in the bucket.
The folders structure is as follows /...
1
vote
1
answer
121
views
Can't delete duplicity s3/glacier archives
For backing up my system to s3, I'm running:
duplicity 0.8.17
Python 3.9.2
boto3 '1.24.36'
My daily backups run fine, as desired they are moved from s3 to glacier after a month, duplicity status ...
0
votes
1
answer
120
views
What IAM policy is appropriate for gitlab S3 buckets when using KMS encryption?
I'm setting up GitLab with S3 storage backend and would like to use KMS to encrypt the buckets. I'd like to avoid giving out permissions broadly. I'm using terraform to provision these resources.
How ...
2
votes
3
answers
1k
views
Is there the equivalent of cloud computing for modems?
I asked this question on SF, and someone recommended that I ask it on SO, and on SO, it was voted to be closed, and someone suggest that I post it here - (and so, the merry go round continues) - so ...
0
votes
1
answer
278
views
How to create AWS S3 key
I've set up an AWS S3 bucket, and it's working for transfers using the web UI, but I want to be able to use the "aws s3 sync" command. The only way I know how to use that command requires ...
1
vote
1
answer
249
views
Is there a way we can way to minimize or eliminate warning when a recipient receives an email from AmazonSES
When we are sending emails using Amazon SES from the application, recipients of the email are seeing warning messages like below:
I have tried to find possible things that we can do to avoid ...
1
vote
1
answer
3k
views
Sonatype Nexus: How to create Blob stores using Amazon S3?
I try to create a blob stores on Sonatype Nexus as below:
Then, I'm getting this error:
Unable to initialize blob store bucket: s3jenkins-cache, Cause: Unable
to execute HTTP request: Connect to
...
2
votes
1
answer
848
views
Can i trigger an AWS lambda function when a sagemaker labeling job is completed using CloudWatch events?
I'm new using CloudWatch events and I'm trying to create an event rule that triggers a lambda function when a Sagemaker Ground Truth Labeling Job is completed.
It seems to me that this is possible, ...
60
votes
5
answers
27k
views
How to automate regular Google Takeout backups to cloud storage
I would like to create regular Google Takeout backups (let's say every 3 months) and store them encrypted in some other cloud storage like DropBox or S3.
It does not have to be a cloud-to-cloud ...
2
votes
1
answer
1k
views
Can WinSCP be used to access Google Cloud Storage bucket using the 'Amazon S3' file protocol?
Can WinSCP be used to access Google Cloud Storage bucket using the Amazon S3 file protocol using the HMAC keys?
I tired but got the following error:
Expected request type in credential scope to be &...
46
votes
8
answers
49k
views
Time machine backup on S3
Is there any possibility to use Amazon S3 for Mac OS X time machine backups?
Or in other words: Use S3 rather an external hard disk for time machine backups?
0
votes
1
answer
42
views
Streaming file via HTTP to S3
Currently we have a function on one of our EC2 instances (running C# .NET) that, upon request, opens an HTTP connection to an external site (for the sake of argument, let's say https://www.example.com/...
5
votes
2
answers
9k
views
How to search for nested files in my S3 bucket?
I'm on the S3 Management Console and I see the instructions to type a prefix to search:
However, when I search, it only finds files in the current folder. Is there a way to search for files that ...
0
votes
1
answer
616
views
Whitelist IP addresses for access to AWS S3 static web hosting
I have seen several references that use a Bucket Policy similar in structure to the following
{
"Version": "2012-10-17",
"Id": "Policy1607646663558",
...
0
votes
0
answers
317
views
Why does s3.amazonaws.com respond with 400 Bad Request when I connect over telnet?
This is what my request looks like. I have confirmed that the end of line separator is CRLF:
telnet s3.amazonaws.com 80
GET / HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; ...
0
votes
0
answers
470
views
Amazon CloudFront buffering while streaming video to Roku
I'm looking for help on trying to troubleshoot a buffering issue using Amazon's CloudFront service. I've set up an S3 bucket and have it associated with a CloudFront service. I use the service to ...
22
votes
5
answers
32k
views
Can I use rsync for backing up at Amazon S3
I have bought some space from evbackup which provides me FTP, rsync and SSH.
But I have also read that Amazon S3 can also be used to backups.
So I am really confused that do Amazon give FTP access ...
1
vote
1
answer
2k
views
Setting up a custom idp for AWS SFTP using Active Directory
I've been digging into setting up custom auth for AWS SFTP and it's one hell of a black hole. There's lot's of references to being able to use a custom idp, but no concrete examples.
We're using ...
0
votes
2
answers
207
views
Windows7 command prompt show number of objects as list from dir command
I'm using Windows7, and I want to know the number of files (or objects) for comparison to objects in Amazon S3 buckets. I'm not interested in size, as the size can differ between file types.
I want ...
1
vote
4
answers
6k
views
How sync files with s3cmd to Amazon S3, check if sent and remove locally?
I'm trying to use Amazon S3 service to store logs from my applications.
Given /user/bin/s3cmd --help tells me what I need to know how send the files:
s3cmd --help
usage: s3cmd [options] COMMAND [...
1
vote
1
answer
2k
views
How can I restore a MySql database dump direct from S3?
I have a gzipped database backup stored on S3 and would like to restore it to MySql without having to first download it due to disk space constraints. I tried the two commands below but got gzip: ...
0
votes
0
answers
182
views
GPG piping help for an S3 encrypted postgresql backup into RDS
I was wondering if there was a trick that I am missing.
I am trying to get an encrypted Postgresql backup dump from S3 -> decrypt it in GPG -> pg_restore into RDS.
However, I am not able to get ...
2
votes
3
answers
2k
views
S4cmd working only for Amazon S3. Is it possible to configure it for DreamObjects cloud storage the way S3cmd worked?
I just switched from S3cmd to S4cmd. It's working great for EC2 to Amazon S3 data transfer. But when I'm trying DreamObjects or CenturyLink Object storage in it's place, it shows the below error:
...
0
votes
1
answer
547
views
Upload multiple files to S3 from stdin
I found the option to upload to S3 from stdin:
cat test.txt | aws s3 cp - s3://mybucket/test.text
But I have a tgz archive coming in via curl and containing multiple files. I want to write the ...
1
vote
1
answer
3k
views
S3FS ownership and permissions
I am trying to mount a S3 bucket to my local machine using secret password key with the following command:
s3fs mybucket /mnt/mybucket -o allow_other -o umask=0000,uid=$UID -o use_cache=/tmp -o ...
0
votes
1
answer
1k
views
Automatic one-way syncing of a Windows file share to an S3 bucket
Our clients have Windows file shares where they store lots of different documents.
We want to ingest files matching wildcard patterns (e.g. *.pdf, *.xlsx, etc.) from those file shares into S3 buckets ...
1
vote
1
answer
335
views
AWS Access Key Issue while accessing s3 bucket on Azure VM
I am trying to access AWS s3 bucket from Azure HDInsight Cluster VM. I generated new keys and added to .aws/credentials. "aws s3 ls" is working fine in Azure VM. If I do hadoop distcp or read a s3 ...
0
votes
0
answers
176
views
Download all S3 files uploaded within a given time period
I would like to all files uploaded to an S3 button within a given time period.
For all example, everything that was uploaded between 05-11-2017 and 07-11-2018.
Please note that due to the size of ...
0
votes
1
answer
4k
views
Selectively uploading multiple folders to AWS S3 using CLI
I have around 50 folders, which contain subfolders, I want to upload the folders and subfolders to an S3 bucket, while keeping the subfolders structure.
I read I can use --include command multiple ...
0
votes
0
answers
14
views
AWS Beanstalk s3 issue
I wanted to update a nodejs app and ruined my s3 server. Instead of updating properly from services -> elastic beanstalk, I deleted the .zip file on s3 and replaced it with a new one. I rebuilt my ...
1
vote
1
answer
11k
views
Get all files (or file names) out of s3 bucket for specific date
I need to get all the files from yesterday from the s3 bucket, i know how to do this in the CLI when i know the file name:
#aws s3 cp s3:{Path}/{FileName} {diretoryToCopyTo}
but how would I do this ...
0
votes
1
answer
64
views
Is there an industry standard tool available which parses s3 files and loads them to database tables?
I have text files of multiple content formats on s3 buckets.
The simplest format had delimited content as dhwon below:
Field1|field2|field3|field4
I would like to parse the above delimited file and ...
0
votes
1
answer
51
views
Prepopulated AWS Access Key ID when creating a new profile?
A new counter-party has provided me with an AWS S3 Access Key ID
and a Secret Key so I can send files to them.
I was expecting to run
$ aws s3 configure
and enter those values when prompted.
...
1
vote
1
answer
70
views
Creating folders by file names
I'm trying to find a way to create a folder for every file in a directory. I then want to copy each file into its created folder. These files are currently hosted on AWS (S3).
So for example if I ...