-1

I have a lot of images, organised into some 40 odd folders, where the largest number in a folder is 1280000 images and the smallest 1 image.

I need to transfer these images to a google cloud compute filestore mounted on a barebones host vm, but to use the gcloud sdk scp commands I've measured it would take roughly 20 hours for 640000 images, so let's call it 12 days for the lot.

This is despite having a 1Gbps synchronous fiber connection to upload from.

Most answers relating to copying large numbers of files, large numbers of small files, large overall amounts of data etc, recommend using rsync to transfer the data, however, there is no gcloud compute command for rsync, the most useful ones in fact are for ssh and scp.

Anyone know a way to transfer these files to a cloud nfs filestore in a way that will more effectively use the available bandwidth? I was thinking tar'ing the files, but that would also take a lot of time to compress and uncompress, right? so wondering how worth it the benefit would be in the end.

2

In the end as unable to make rsync behave from the filestore vm pulling the images from my local boxes, I gave up and resorted to:

ssh user@local.box 'tar -cf - -C /local/box/file/location .' | tar -xvf - -C /mnt/google/filestore/box/location

to just pipe the files without compression.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.