Questions tagged [amazon-s3]

Amazon S3 (Simple Storage Service) is a web service which provides storage and various interfaces for accessing data stored there.

Filter by
Sorted by
Tagged with
0 votes
1 answer
16 views

Connect to S3 Web Console with Access and Secret Key

Is it possible to authenticate in the Amazon S3 console (https://s3.console.aws.amazon.com) using an access key/secret key pair? It looks like it only accepts user credentials. If not, is there any ...
mrgou's user avatar
  • 233
0 votes
0 answers
48 views

AWS S3 <-> MS Onedrive sync with rules

Is it possible to have an automatic synchronization between a specific AWS S3 bucket and a OneDrive folder ? and to specify rules e.g. remove certain files extensions from the sync ; only sync ...
Nicolas Rosewick's user avatar
1 vote
0 answers
21 views

AWS Windows EC2 File Transfer - Need to keep session open

I have a rather large file that I need to copy from an S3 bucket to a Windows EC2 and it will not complete before a session ends. I have found lots of suggestions for this for Linux but not Windows. ...
kcjelv178's user avatar
0 votes
1 answer
223 views

aws sync doesn't download the file and instead create an empty directory

The file I'm trying to download is: s3://ont-open-data/gm24385_q20_2021.10/analysis/20210805_1713_5C_PAH79257_0e41e938/guppy_5.0.15_sup/align_unfiltered/chr19/calls2ref.bam This is an open dataset. I ...
Jerry Fan's user avatar
0 votes
1 answer
85 views

How translate s3 url to http in chrome automatically?

Let say I have a link like that: s3://mybucket/mybucket?prefix=myprefix/file.json And I open it in chrome browser. I will get an error. Is it possible to automatically translate the s3 url from ...
Cherry's user avatar
  • 870
1 vote
0 answers
190 views

AWS S3 Bucket CORS Error

I have an amazon s3 bucket called cs-ia. This is my CORS configuration: [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ ...
Berlm's user avatar
  • 111
1 vote
1 answer
86 views

Can't delete duplicity s3/glacier archives

For backing up my system to s3, I'm running: duplicity 0.8.17 Python 3.9.2 boto3 '1.24.36' My daily backups run fine, as desired they are moved from s3 to glacier after a month, duplicity status ...
htInEdin's user avatar
0 votes
1 answer
704 views

make new objects in s3 bucket public by default

I have an s3 bucket in which the objects within it need to be made public, right now anytime I upload a new file I have to click on it and then click "make public using acl" I wonder if ...
fred's user avatar
  • 113
0 votes
1 answer
88 views

What IAM policy is appropriate for gitlab S3 buckets when using KMS encryption?

I'm setting up GitLab with S3 storage backend and would like to use KMS to encrypt the buckets. I'd like to avoid giving out permissions broadly. I'm using terraform to provision these resources. How ...
flickerfly's user avatar
  • 1,001
0 votes
1 answer
71 views

How to create AWS S3 key

I've set up an AWS S3 bucket, and it's working for transfers using the web UI, but I want to be able to use the "aws s3 sync" command. The only way I know how to use that command requires ...
Toby Eggitt's user avatar
1 vote
1 answer
201 views

Is there a way we can way to minimize or eliminate warning when a recipient receives an email from AmazonSES

When we are sending emails using Amazon SES from the application, recipients of the email are seeing warning messages like below: I have tried to find possible things that we can do to avoid ...
BunnY's user avatar
  • 11
2 votes
1 answer
1k views

Can WinSCP be used to access Google Cloud Storage bucket using the 'Amazon S3' file protocol?

Can WinSCP be used to access Google Cloud Storage bucket using the Amazon S3 file protocol using the HMAC keys? I tired but got the following error: Expected request type in credential scope to be &...
Sarang's user avatar
  • 21
0 votes
1 answer
37 views

Streaming file via HTTP to S3

Currently we have a function on one of our EC2 instances (running C# .NET) that, upon request, opens an HTTP connection to an external site (for the sake of argument, let's say https://www.example.com/...
JED's user avatar
  • 101
0 votes
1 answer
561 views

Whitelist IP addresses for access to AWS S3 static web hosting

I have seen several references that use a Bucket Policy similar in structure to the following { "Version": "2012-10-17", "Id": "Policy1607646663558", ...
WestCoastProjects's user avatar
0 votes
0 answers
266 views

Why does s3.amazonaws.com respond with 400 Bad Request when I connect over telnet?

This is what my request looks like. I have confirmed that the end of line separator is CRLF: telnet s3.amazonaws.com 80 GET / HTTP/1.1 Host: s3.amazonaws.com User-Agent: Mozilla/5.0 (Windows NT 10.0; ...
PHP Guru's user avatar
  • 101
0 votes
2 answers
202 views

Windows7 command prompt show number of objects as list from dir command

I'm using Windows7, and I want to know the number of files (or objects) for comparison to objects in Amazon S3 buckets. I'm not interested in size, as the size can differ between file types. I want ...
Jordan's user avatar
  • 1
0 votes
0 answers
154 views

GPG piping help for an S3 encrypted postgresql backup into RDS

I was wondering if there was a trick that I am missing. I am trying to get an encrypted Postgresql backup dump from S3 -> decrypt it in GPG -> pg_restore into RDS. However, I am not able to get ...
Jonathan's user avatar
  • 101
0 votes
1 answer
474 views

Upload multiple files to S3 from stdin

I found the option to upload to S3 from stdin: cat test.txt | aws s3 cp - s3://mybucket/test.text But I have a tgz archive coming in via curl and containing multiple files. I want to write the ...
Marcel Boldt's user avatar
4 votes
2 answers
7k views

How to search for nested files in my S3 bucket?

I'm on the S3 Management Console and I see the instructions to type a prefix to search: However, when I search, it only finds files in the current folder. Is there a way to search for files that ...
nachocab's user avatar
  • 147
1 vote
1 answer
288 views

AWS Access Key Issue while accessing s3 bucket on Azure VM

I am trying to access AWS s3 bucket from Azure HDInsight Cluster VM. I generated new keys and added to .aws/credentials. "aws s3 ls" is working fine in Azure VM. If I do hadoop distcp or read a s3 ...
rajdrm's user avatar
  • 11
0 votes
0 answers
144 views

Download all S3 files uploaded within a given time period

I would like to all files uploaded to an S3 button within a given time period. For all example, everything that was uploaded between 05-11-2017 and 07-11-2018. Please note that due to the size of ...
Blunderchips's user avatar
0 votes
1 answer
3k views

Selectively uploading multiple folders to AWS S3 using CLI

I have around 50 folders, which contain subfolders, I want to upload the folders and subfolders to an S3 bucket, while keeping the subfolders structure. I read I can use --include command multiple ...
Tlink's user avatar
  • 169
0 votes
0 answers
13 views

AWS Beanstalk s3 issue

I wanted to update a nodejs app and ruined my s3 server. Instead of updating properly from services -> elastic beanstalk, I deleted the .zip file on s3 and replaced it with a new one. I rebuilt my ...
Georgios Lytos's user avatar
0 votes
1 answer
51 views

Prepopulated AWS Access Key ID when creating a new profile?

A new counter-party has provided me with an AWS S3 Access Key ID and a Secret Key so I can send files to them. I was expecting to run $ aws s3 configure and enter those values when prompted. ...
zundarz's user avatar
  • 1,155
1 vote
1 answer
46 views

Creating folders by file names

I'm trying to find a way to create a folder for every file in a directory. I then want to copy each file into its created folder. These files are currently hosted on AWS (S3). So for example if I ...
BobDoleForPresident's user avatar
1 vote
1 answer
2k views

How to configure amazon S3 bucket for read-only access to host a static website?

I have a very small (static) website hosted on Amazon S3. Yesterday I received the following message, that had this subject line: Notification of Amazon S3 buckets configured for public access And ...
c00000fd's user avatar
  • 517
1 vote
1 answer
2k views

Setting up a custom idp for AWS SFTP using Active Directory

I've been digging into setting up custom auth for AWS SFTP and it's one hell of a black hole. There's lot's of references to being able to use a custom idp, but no concrete examples. We're using ...
AddGarbanzos's user avatar
1 vote
1 answer
2k views

Sonatype Nexus: How to create Blob stores using Amazon S3?

I try to create a blob stores on Sonatype Nexus as below: Then, I'm getting this error: Unable to initialize blob store bucket: s3jenkins-cache, Cause: Unable to execute HTTP request: Connect to ...
Cường Phạm's user avatar
0 votes
0 answers
66 views

AWS S3 Upload Broken Up

perhaps this is my own misunderstanding of how AWS CLI works, but I uploaded a converted qcow2 VM image converted to a .raw using the AWS CLI. It took about 6 hours to move the 128GB file. I let it ...
stfuaboutmoonmen's user avatar
0 votes
1 answer
53 views

Is there an industry standard tool available which parses s3 files and loads them to database tables?

I have text files of multiple content formats on s3 buckets. The simplest format had delimited content as dhwon below: Field1|field2|field3|field4 I would like to parse the above delimited file and ...
biju_c's user avatar
  • 11
2 votes
0 answers
255 views

How to use AWS S3 for linux file system backup?

As many who have used AWS S3 realize, it isn't a file system, but everything in the bucket is an object. While using the aws cli to do backups of a Linux server to S3, it doesn't preserve the ...
Edward_178118's user avatar
1 vote
1 answer
2k views

I am trying to use a kms key alias in a bucket policy

I am using the following bucket policy: { "Version": "2012-10-17", "Id": "Policy1566337631537", "Statement": [ { "Sid": "Stmt156", "Effect": "Deny", ...
jmalacho's user avatar
1 vote
1 answer
9k views

Get all files (or file names) out of s3 bucket for specific date

I need to get all the files from yesterday from the s3 bucket, i know how to do this in the CLI when i know the file name: #aws s3 cp s3:{Path}/{FileName} {diretoryToCopyTo} but how would I do this ...
Vincent's user avatar
  • 111
0 votes
0 answers
189 views

Lost my files while moving on AWS S3

I used the command on CLI - aws s3 mv s3://<source_directory_path> <destination_directory_path> --recursive aws s3 mv s3://<source_directory_path> s3://<...
Hardik Parikh's user avatar
2 votes
1 answer
714 views

Can i trigger an AWS lambda function when a sagemaker labeling job is completed using CloudWatch events?

I'm new using CloudWatch events and I'm trying to create an event rule that triggers a lambda function when a Sagemaker Ground Truth Labeling Job is completed. It seems to me that this is possible, ...
José R. Sánchez's user avatar
0 votes
1 answer
183 views

Cannot seem to access s3 bucket for training after mounting onto ec2 instance

I am using data on a s3 bucket for deep learning purposes. I mounted the s3 bucket onto ec2 instance using How to Mount S3 bucket on EC2 Linux Instance - Cloudkul. When I cd into my local directory ...
Ping's user avatar
  • 1
0 votes
1 answer
500 views

Source url prefix appended to destination url when using CloudFront and S3 AWS

I am trying to redirect a url, example.io, to www.example.org/survey/imagine. Currently, my S3 bucket handles redirection with the following redirection rules: <RoutingRules> <RoutingRule&...
Adam Tapp's user avatar
0 votes
1 answer
268 views

Serving tar.gz file through AWS Cloudfront?

I would like to keep things tidy on the S3 bucket where my website's assets live. Currently I am uploading all assets to S3, as part of the CI/CD pipeline, this has got a bit messy since generated ...
Carlos Valencia's user avatar
0 votes
0 answers
487 views

Bash completion not working for 'aws cli' after installed via brew

After I use brew to install aws cli, here is the message form brew: ==> awscli The "examples" directory has been installed to: /usr/local/share/awscli/examples Bash completion has been ...
Anthony Kong's user avatar
  • 4,898
1 vote
1 answer
2k views

S3FS ownership and permissions

I am trying to mount a S3 bucket to my local machine using secret password key with the following command: s3fs mybucket /mnt/mybucket -o allow_other -o umask=0000,uid=$UID -o use_cache=/tmp -o ...
Buggy B's user avatar
  • 111
0 votes
1 answer
1k views

AWS ACCESS KEY ISSUE

I accidentally pushed my access keys up to github and amazon contacted me with instructions on "fixing" the issue. I messed up and accidentally deleted the access key, thinking thats what i was ...
isaiah santillan's user avatar
1 vote
1 answer
84 views

How do I set up DNS, AWS S3, AWS Cloudfront, and AWS Certificate Manager to secure multiple sites

I am trying to secure multiple static AWS S3 websites delivered via AWS Cloudfront CDN with one certificate. This AWS article says I should be able to do this, but it's not working. One is a web-app,...
LWSChad's user avatar
  • 111
3 votes
1 answer
971 views

Nginx SSL Preread Sporadically Gets Requests Where No Server Name Is Extracted

I am using the nginx stream module to leverage nginx as a tcp reverse proxy in front of s3. Recently, I needed to add logic to accommodate an additional upstream. To do this, I chose to make use of ...
Howard_Roark's user avatar
0 votes
1 answer
985 views

Automatic one-way syncing of a Windows file share to an S3 bucket

Our clients have Windows file shares where they store lots of different documents. We want to ingest files matching wildcard patterns (e.g. *.pdf, *.xlsx, etc.) from those file shares into S3 buckets ...
Gui Prá's user avatar
  • 133
7 votes
1 answer
5k views

Know the identity behind an AWS canonical ID

Today while auditing a friends security settings, I found that an Amazon Folder (s3 bucket) with his most personal files had an unrecognised ACL entry. A person (with id ending in f6995f) had "Write" ...
american-ninja-warrior's user avatar
2 votes
0 answers
519 views

AWS S3 CLI: sum bytes in all top-level prefixes with a single --query?

I have an S3 bucket with a bunch of top-level prefixes. I have a relatively inefficient way of summing the bytes across top-level prefixes: Get the list of prefixes via aws s3 ls [bucket name], ...
Steve Shulman-Laniel's user avatar
0 votes
1 answer
710 views

How to auto upload a file to s3

How can I upload a json file to my s3 in easiest way? Basically there is a json file in a webpage and I want to keep copy it every 5 minutes and upload to my s3.
Tel4tel's user avatar
0 votes
1 answer
1k views

Linux - How to disable disk cache for a specific mount point?

Is it possible to disable disk cache for a mount point on Linux system? I have a s3fs mount on multiple systems and i need to disable disk cache for it. how can i achieve this?
Mozafar Gholami's user avatar
0 votes
0 answers
591 views

goofys mount point on multiple linux systems

I have a Minio bucket which is mounted on three different linux systems. the problem is when a file changes on one of mount points but file size doesn't changes that change doesn't get reflected on ...
Mozafar Gholami's user avatar
2 votes
1 answer
6k views

How to fix awscli s3 error "The Content-MD5 you specified did not match what we received" (BadDigest)?

Uploading a file over 10 GB to s3 fails with: aws s3 cp 'foo.log.gz' s3://mybucket/2018/12/ --storage-class REDUCED_REDUNDANCY upload failed: ./foo.log.gz to s3://mybucket/2018/12/foo.log.gz An ...
user51070's user avatar
  • 135