0

So I want to copy a 97,9GB folder to another folder and started with a high transfer rate and an estimated time of about 30 minutes. As the fdsr continues to copy the transfer rate goes down. Its at about 5Mb/Sec now and the estimated time raised to 5 Hours. The Rate is still dropping (while I wrote this its at 4,3Mb/s already) how can I avoid this and copy the folder at a consistent high transfer rate? I am using ubuntu

2
  • Are you sure you mean Mb/sec instead of MB/sec? How many files, and specifically how many small files do you have? Have you tried tarring up the source folder, transferring the tar file, and untarring on the destination side? Sep 26, 2015 at 14:33
  • Yes its MB/second and there are about 500,000 Files to be copied. I figured out that copying everything in small portions results in a faster total time. I am still Interested why this is happening. Sep 26, 2015 at 14:35

1 Answer 1

0

In general, it is much faster to copy larger files than many smaller files. When transferring a single large file, your copy time is dominated by transfer speed. When copying many small files, your time is dominated by file creation.

You could probably speed things up by tarring up your source folder and transferring the tar file, but note if you have 500,000 files to transfer, you'll have to pay the cost of all that metadata creation sooner or later.

You may also find different file systems are more or less costly. https://unix.stackexchange.com/questions/28756/what-is-the-most-high-performance-linux-filesystem-for-storing-a-lot-of-small-fi is quite old now, but includes a comparison. Additionally, you may choose different mount options for your filesystem, as some of these can make a significant difference. For example, on ext4, you may wish to investigate noatime,nodiratime,relatime,barrier=0,commit=30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.