Fastest way to process and transfer around 600GB of data between two external hard disks

Question

sudo fdisk -l :

will give me this following output.

Disk /dev/sdc: 4000.8 GB, 4000787029504 bytes
256 heads, 63 sectors/track, 484501 cylinders, total 7814037167 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 33553920 bytes
Disk identifier: 0x1ca6469a

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1  4294967295  2147483647+  ee  GPT
Partition 1 does not start on physical sector boundary.

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

Note: sector size is 4096 (not 512)

Disk /dev/sdd: 3000.6 GB, 3000592977920 bytes
255 heads, 63 sectors/track, 45600 cylinders, total 732566645 sectors
Units = sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1               1       51205      204820   ee  GPT
/dev/sdd2           51206   732533871  2929930664    7  HPFS/NTFS/exFAT

I want to transfer data from device mounted on /dev/sdd2 to /dev/sdc2 and data is around 400 - 600GB. What is the best method to processs and transfer this file.

My destination disk is not empty and I need to move this amount of data every three days.

1. Zipping and unzipping takes too much time
2. I am yet to try rsync, would like your reviews
3. I am using 2 USB 3.0 ports for both the hard disks. Should I NOT connect them to a hub to make it faster?

Is there any other efficient way of transferring these files?

Answer 1

The best way will be a straight copy - unless there is already data on the destination drive which matches the source drive with identical filename, size and last mode date you are going to be wasting resources. This is even truer of using compression / decompression unless you are doing their data compressed - and the source data is compressable (and assuming reads are faster then write)

You don't want to use a hub as this will share the hard disk bandwidth.

You may be able to tweek things round the edges by changing the filesystem block size - larger is more wasteful but slightly faster. (Not massively though)

Also, different filesystems could give you better performance depending on the file size.

Your bottleneck is really likely to be the disk speed and latency more then anything else. Not much you can do to fix this, save, perhaps enduring the target disk is blank - as the initial - outermost tracks - are a lot faster then the end - innermost tracks. (I assume your reference to sdc2 should have been sdc1. If that's NOT the case, you should consider sdc's layout with respect of putting data as close to start of disk as possible.

Answer 2

If you have two equivalent USB hubs, connect the disks to different hubs, disconnect other USB devices and do a straight copy. This will ensure maximum parallelism.

If you only have one hub, it may be faster to transfer to an intermediate internal disk, then copy to the other external disk, as the external disk will be the bottleneck.

In case you have folders with lots of small files, compressing them in one zip archive will increase transfer speed dramatically.

When drives are the same size then Drive-to-Drive partition copy should be fastest way if the disk is mostly full, but slower if the disk is far from full.

If the destination drive is empty, increasing the partition cluster size from 4 to 16 will prevent a lot of overhead I/O operations, but cost some extra space.