Questions tagged [amazon-s3]
Amazon S3 (Simple Storage Service) is a web service which provides storage and various interfaces for accessing data stored there.
169
questions
0
votes
0answers
14 views
Cannot write into mounted AWS SFTP
I have recently created a server under the AWS Transfer Family to access my S3 bucket via SFTP. All the security parameter have been setup (keys and policies) and I am unable to write data to my ...
0
votes
0answers
40 views
PowerCmd does not print out aws commands' outputs
aws commands works well with cmd.exe. But the outputs are not shown when I use it with Powercmd.
I've attached one example.
It seems that Powercmd is not throwing any error. I can enter credentials ...
0
votes
0answers
17 views
Cyberduck view thumbnails of S3 bucket full of images?
Is it possible using Cyberduck or another tool to preview thumbnails of a s3 bucket full of images (jpg, etc)
0
votes
0answers
14 views
Download a file and copy to another server without downloading on server
I have a Postgres RDS, Ubuntu EC2 and S3. I want to download a table in SQL dump and upload it to S3.
In my RDS, I have a table which is around 2000 GB big. My EC2 only has 500GB of storage left.
To ...
0
votes
1answer
34 views
Upload multiple files to S3 from stdin
I found the option to upload to S3 from stdin:
cat test.txt | aws s3 cp - s3://mybucket/test.text
But I have a tgz archive coming in via curl and containing multiple files. I want to write the ...
0
votes
0answers
8 views
Upload to S3 from small container takes too long time
I have a container which is running a process that outputs files. Those files is then uploaded to S3 by using the aws s3 cp command. The files is about 325 kb and it takes about 4.5 seconds to upload ...
0
votes
1answer
126 views
How to search for nested files in my S3 bucket?
I'm on the S3 Management Console and I see the instructions to type a prefix to search:
However, when I search, it only finds files in the current folder. Is there a way to search for files that ...
1
vote
1answer
64 views
AWS Access Key Issue while accessing s3 bucket on Azure VM
I am trying to access AWS s3 bucket from Azure HDInsight Cluster VM. I generated new keys and added to .aws/credentials. "aws s3 ls" is working fine in Azure VM. If I do hadoop distcp or read a s3 ...
0
votes
0answers
21 views
Download all S3 files uploaded within a given time period
I would like to all files uploaded to an S3 button within a given time period.
For all example, everything that was uploaded between 05-11-2017 and 07-11-2018.
Please note that due to the size of ...
0
votes
0answers
14 views
How to make non-tech savvy users download entire S3 bucket?
I've got 200GB of data uploaded to an S3 bucket, and I've got some none tech savvy (so no answers involving shells, SSH keys, CLIs etc or anything average Joe wouldn't understand) users to download ...
0
votes
1answer
226 views
Selectively uploading multiple folders to AWS S3 using CLI
I have around 50 folders, which contain subfolders, I want to upload the folders and subfolders to an S3 bucket, while keeping the subfolders structure.
I read I can use --include command multiple ...
0
votes
0answers
7 views
How to use minio s3 as download link in apache server, with expiry tag?
We have configured minio S3 cluster, and we need to host on premise web server with minio s3 as storage back end.
We chose "apache" as web server
I was went through minio docs and didnt find any ...
0
votes
0answers
7 views
AWS Beanstalk s3 issue
I wanted to update a nodejs app and ruined my s3 server. Instead of updating properly from services -> elastic beanstalk, I deleted the .zip file on s3 and replaced it with a new one. I rebuilt my ...
0
votes
1answer
29 views
Prepopulated AWS Access Key ID when creating a new profile?
A new counter-party has provided me with an AWS S3 Access Key ID
and a Secret Key so I can send files to them.
I was expecting to run
$ aws s3 configure
and enter those values when prompted.
...
1
vote
1answer
24 views
Creating folders by file names
I'm trying to find a way to create a folder for every file in a directory. I then want to copy each file into its created folder. These files are currently hosted on AWS (S3).
So for example if I ...
0
votes
1answer
715 views
How to configure amazon S3 bucket for read-only access to host a static website?
I have a very small (static) website hosted on Amazon S3. Yesterday I received the following message, that had this subject line:
Notification of Amazon S3 buckets configured for public access
And ...
1
vote
1answer
758 views
Setting up a custom idp for AWS SFTP using Active Directory
I've been digging into setting up custom auth for AWS SFTP and it's one hell of a black hole. There's lot's of references to being able to use a custom idp, but no concrete examples.
We're using ...
1
vote
0answers
339 views
Sonatype Nexus: How to create Blob stores using Amazon S3?
I try to create a blob stores on Sonatype Nexus as below:
Then, I'm getting this error:
Unable to initialize blob store bucket: s3jenkins-cache, Cause: Unable
to execute HTTP request: Connect to
...
0
votes
0answers
20 views
AWS S3 Upload Broken Up
perhaps this is my own misunderstanding of how AWS CLI works, but I uploaded a converted qcow2 VM image converted to a .raw using the AWS CLI. It took about 6 hours to move the 128GB file. I let it ...
0
votes
0answers
30 views
Can Amazon DMS migrate data from S3 delimited files to MS SQL Server tables?
The DMS home page (https://aws.amazon.com/dms/) says:
"With AWS Database Migration Service, you can continuously replicate your data with high availability and consolidate databases into a petabyte-...
0
votes
1answer
44 views
Is there an industry standard tool available which parses s3 files and loads them to database tables?
I have text files of multiple content formats on s3 buckets.
The simplest format had delimited content as dhwon below:
Field1|field2|field3|field4
I would like to parse the above delimited file and ...
2
votes
0answers
51 views
How to use AWS S3 for linux file system backup?
As many who have used AWS S3 realize, it isn't a file system, but everything in the bucket is an object. While using the aws cli to do backups of a Linux server to S3, it doesn't preserve the ...
1
vote
1answer
607 views
I am trying to use a kms key alias in a bucket policy
I am using the following bucket policy:
{
"Version": "2012-10-17",
"Id": "Policy1566337631537",
"Statement": [
{
"Sid": "Stmt156",
"Effect": "Deny",
...
0
votes
0answers
314 views
Write YARN aggregated log to S3 on AWS EMR
I'm using emr-5.26.0 and I followed the documentation here (in the 'To aggregate logs in Amazon S3 using the AWS CLI' section) and I was not able to have the aggregated log file from the my spark app.
...
1
vote
1answer
1k views
Get all files (or file names) out of s3 bucket for specific date
I need to get all the files from yesterday from the s3 bucket, i know how to do this in the CLI when i know the file name:
#aws s3 cp s3:{Path}/{FileName} {diretoryToCopyTo}
but how would I do this ...
0
votes
0answers
20 views
How can I access ProFTPD, if XAMPP to AWS web server was done completely through the CLI?
I created a web server using EC2 and LAMPP. I can access the myPHPadmin, but I have no idea how to access ProFTPD. Additionally, XAMPP did not download locally on my pc so I cannot access htdocs. The ...
0
votes
0answers
74 views
How do I copy from an S3 to an Azure Storage File Share?
I see instructions on how to copy from S3 to Azure Blob using AzCopy https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-s3#copy-objects-directories-and-buckets
But I need to use ...
0
votes
0answers
43 views
Lost my files while moving on AWS S3
I used the command on CLI -
aws s3 mv s3://<source_directory_path> <destination_directory_path> --recursive
aws s3 mv s3://<source_directory_path> s3://<...
1
vote
0answers
199 views
Can i trigger an AWS lambda function when a sagemaker labeling job is completed using CloudWatch events?
I'm new using CloudWatch events and I'm trying to create an event rule that triggers a lambda function when a Sagemaker Ground Truth Labeling Job is completed.
It seems to me that this is possible, ...
0
votes
0answers
17 views
Skip Ahead in AWS Console
In the AWS S3 console (https://s3.console.aws.amazon.com/s3/buckets) you can only go forwards or backwards one page at a time. When you have hundreds of thousands of files this becomes untenable. Is ...
0
votes
1answer
49 views
Cannot seem to access s3 bucket for training after mounting onto ec2 instance
I am using data on a s3 bucket for deep learning purposes. I mounted the s3 bucket onto ec2 instance using How to Mount S3 bucket on EC2 Linux Instance - Cloudkul.
When I cd into my local directory ...
0
votes
1answer
156 views
Source url prefix appended to destination url when using CloudFront and S3 AWS
I am trying to redirect a url, example.io, to www.example.org/survey/imagine.
Currently, my S3 bucket handles redirection with the following redirection rules:
<RoutingRules>
<RoutingRule&...
0
votes
1answer
49 views
Serving tar.gz file through AWS Cloudfront?
I would like to keep things tidy on the S3 bucket where my website's assets live.
Currently I am uploading all assets to S3, as part of the CI/CD pipeline, this has got a bit messy since generated ...
0
votes
0answers
176 views
Bash completion not working for 'aws cli' after installed via brew
After I use brew to install aws cli, here is the message form brew:
==> awscli
The "examples" directory has been installed to:
/usr/local/share/awscli/examples
Bash completion has been ...
0
votes
1answer
321 views
S3FS ownership and permissions
I am trying to mount a S3 bucket to my local machine using secret password key with the following command:
s3fs mybucket /mnt/mybucket -o allow_other -o umask=0000,uid=$UID -o use_cache=/tmp -o ...
0
votes
0answers
248 views
AWS s3 cross account policy allowing public access
I'm having some difficulty locking down an S3 bucket that I must share across accounts.
I have no public ACL defined. "Everyone" has no options enabled.
The console bucket list says "Bucket and ...
0
votes
1answer
470 views
AWS ACCESS KEY ISSUE
I accidentally pushed my access keys up to github and amazon contacted me with instructions on "fixing" the issue. I messed up and accidentally deleted the access key, thinking thats what i was ...
1
vote
1answer
51 views
How do I set up DNS, AWS S3, AWS Cloudfront, and AWS Certificate Manager to secure multiple sites
I am trying to secure multiple static AWS S3 websites delivered via AWS Cloudfront CDN with one certificate.
This AWS article says I should be able to do this, but it's not working.
One is a web-app,...
2
votes
1answer
282 views
Nginx SSL Preread Sporadically Gets Requests Where No Server Name Is Extracted
I am using the nginx stream module to leverage nginx as a tcp reverse proxy in front of s3. Recently, I needed to add logic to accommodate an additional upstream. To do this, I chose to make use of ...
0
votes
1answer
402 views
Automatic one-way syncing of a Windows file share to an S3 bucket
Our clients have Windows file shares where they store lots of different documents.
We want to ingest files matching wildcard patterns (e.g. *.pdf, *.xlsx, etc.) from those file shares into S3 buckets ...
0
votes
1answer
1k views
Know the identity behind an AWS canonical ID
Today while auditing a friends security settings, I found that an Amazon Folder (s3 bucket) with his most personal files had an unrecognised ACL entry.
A person (with id ending in f6995f) had "Write" ...
2
votes
0answers
254 views
AWS S3 CLI: sum bytes in all top-level prefixes with a single --query?
I have an S3 bucket with a bunch of top-level prefixes. I have a relatively inefficient way of summing the bytes across top-level prefixes:
Get the list of prefixes via aws s3 ls [bucket name], ...
0
votes
1answer
388 views
How to auto upload a file to s3
How can I upload a json file to my s3 in easiest way?
Basically there is a json file in a webpage and I want to keep copy it every 5 minutes and upload to my s3.
0
votes
1answer
333 views
Linux - How to disable disk cache for a specific mount point?
Is it possible to disable disk cache for a mount point on Linux system?
I have a s3fs mount on multiple systems and i need to disable disk cache for it. how can i achieve this?
0
votes
0answers
205 views
goofys mount point on multiple linux systems
I have a Minio bucket which is mounted on three different linux systems. the problem is when a file changes on one of mount points but file size doesn't changes that change doesn't get reflected on ...
0
votes
1answer
2k views
How to fix awscli s3 error “The Content-MD5 you specified did not match what we received” (BadDigest)?
Uploading a file over 10 GB to s3 fails with:
aws s3 cp 'foo.log.gz' s3://mybucket/2018/12/ --storage-class REDUCED_REDUNDANCY
upload failed: ./foo.log.gz to s3://mybucket/2018/12/foo.log.gz
An ...
0
votes
3answers
3k views
AWS S3 downloaded files have “last modified date” of the file in S3 after download
I'm using AWS CLI to download some files from S3 (command: 'aws s3 sync'). The problem I have is, that the downloaded file has the same "last modified" date as the file in S3. What I want is that the "...
1
vote
2answers
3k views
copy specific files from s3 bucket
I have some files in s3 bucket.
aws s3 ls s3://bucketname/
file-111-100x100.jpg
file-112-1400x1400.jpg
file-123-250x250.jpg
file-231-1400x1400.jpg
file-222-700x700.jpg
file-333-100x100.jpg
file-131-...
0
votes
2answers
130 views
Why CNAME is not working as supposed?
I have set a CNAME entry to redirect over Amazon S3 bucket, however there is something that doesnt works as supposed (by me of course) with CNAME I expect to redirect a fictional subdomain, say for ...
1
vote
2answers
6k views
curl can not download file but browser can
URL example: https://xxxxxxxxxx.s3.amazonaws.com/xxxx-xxxx-xxxx-xxxx/xxxxxxxxxxxxx/x?AWSAccessKeyId=AKIAIFKEBR4CLVTDYSLA&Signature=Kv%2B859dJ2eUmLgHdkT%2F9nUJEeRM%3D&Expires=1922517562
I ...
