Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've stared mirroring the files and directories on an FTP server to my external HDD using my Raspberry Pi so I can leave the downloads running 24/7.

What I did to start the download was:

lftp user@adress.com

mirror --use-pget-n=8 /

The files are downloading, however it seems to be running quite slow. After about 10 hours of running, it's only downloaded 139GB. Whenever I ran the download using Filezilla on my desktop (Mint) I got ~200GB in 4 hours.

I was unaware of how many segments each file should be downloaded in, so I choose 8. To be honest, I'm not entirely sure of the benefit splitting the file into segments.

My download speed according to Speedtest.net peaks at around 40Mb/sec and is usually around 30Mb/sec.

Are there any better parameters I could use, or is it possibly down the Raspberry Pi's hardware?

I would tar all the files and download them in one go, only I don't have any other access to the server (not the owner).

Thanks.

share|improve this question
6  
Is there a specific reason you don't want rsync? –  terdon Jan 16 '13 at 22:14
1  
I don't have any other access other than FTP as I don't own the server. The files are all public, but I only have access via FTP. –  Nysepho Jan 16 '13 at 22:31
1  
Ah, OK. I would first compare the speed of the desktop and RPi by downloading the same file a few times. I mean run the same FTP command on both machines. That way you'll know if it is a hardware limitation. Also, some of the programs suggested on this thread. might help. –  terdon Jan 16 '13 at 22:41
add comment

1 Answer

up vote 1 down vote accepted

The fastest way do download the files from that FTP server would be to run lftp on the server and download the files over the loopback interface.

Your question is imprecise (leave that aside for now).


Speedtest gives bits per second. 40 Mbit/s is 5 MByte. 5 MB/s * 3600 sec * 4 = 72k MB or roughly 70 Giga bytes. If your speed was 40 MB/s (320 Mbit/s), then you would have downloaded roughly 560 giga bytes in four hours. I don't think even the newest consumer grade hard drives can write that fast, so you would need an SSD drive. Are those even available in 500 or more gigs?

You are seeing a bottleneck of 47 MB/sec with the described configuration and the most likely source of that is (if not the hard drive itself) the USB connection of the external hard drive. However, you did not state that the drive was connected via USB. The RasPi could be downloading to a network drive, for all I know. In that case, you would still be limited by the USB 2 theoretical 60MB/s limit because the ethernet network on the RasPi is actually a USB adapter.

My comments are inconclusive, but I think it's reasonable to conclude that the RasPi hardware is to blame for the bottleneck.


File segmentation becomes more important when you are downloading over an unstable connection (or UDP). If your net does not frequently fall out while one file is being transferred, using application level segmentation will give you nothing but a few extra check sums. Actually, in all likelihood, you will not even see those.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.