0

I have around 50 folders, which contain subfolders, I want to upload the folders and subfolders to an S3 bucket, while keeping the subfolders structure.

  1. I read I can use --include command multiple times, but is there a better way to do that?
  2. The folders that I want to upload are created after a certain date, is there a way to pipe the date range into the command?

thanks

Improve this question

1 Answer 1

Reset to default
1

Command Line

aws s3 cp (and sync) is well documented. The primary options are include or exclude, which lets you use filters include and exclude by folder or file type. Some examples from the documentation

aws s3 cp /tmp/foo s3://bucket/ --recursive --exclude ".git/*"

aws s3 cp /tmp/foo s3://bucket/ --recursive --include "*.txt" --exclude "*"

You may find the --recursive flag useful as well.

If you want to upload based on date I think you're going to have to write a script or code to do it. A quick Google search found this answer which is relevant.

Easier Way

I use CloudBerry Backup (recently renamed MSP360) for backing up my PC to S3. It has a nice GUI to select folders to back up, or optionally you can configure the backup plan with the option "Backup files modified since".

There's a free trial, after that it's $50, which is pretty cheap given the time it could save you.

Cloudberry Backup

Improve this answer
4
  • thanks @Tim for the answer, I was avoiding using --include and having to type 50 long folder names. Folder sizes are long each folder will take some time to upload. I was hoping that there would be a CLI command that pipes files filtered by a date range. I'm not sure if I write a Python script that the script will have to do the authentication independent of the aws cli authentication, authenticating aws programmatically is usually complicate when not using their language specific SDK.
    – Tlink
    Oct 31, 2019 at 1:58
  • You could possibly do it using a bash script and the API, but that's beyond what I know. Also, I wonder if there's a command line limit length. In your place I'd probably be looking for a tool / script someone else already wrote that could do something like this. Could you filter by extension?
    – Tim
    Oct 31, 2019 at 7:28
  • I suspect this is possible s3express.com but it will only work for 21 days, after that you have to pay for it. I haven't quite worked it out, but look at the comments on this page.
    – Tim
    Oct 31, 2019 at 7:41
  • I added another option to the answer - CloudBerry backup. I use it myself, but I haven't used this feature, but the software in general is good.
    – Tim
    Oct 31, 2019 at 7:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .