Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'd like to know why technically Dropbox is much faster than FTP? What kind of technology does it use?

I'm not talking about diff files, I'm talking about transferring new files in both cases, Dropbox is much faster.

I mean it, very much faster, maybe 10 times faster than FTP for files I uploaded. I will experiment again for bigger files later.

share|improve this question
2  
What size, type, and number of files did you upload? How long did each of them take to upload? Where were you uploading the files to via FTP? Dropbox is not magic, the simplest explanation is that the FTP server you were uploading too has much less bandwidth than Amazon does. –  user23307 Mar 15 '10 at 1:50
2  
if they already have it, it dosen't re-upload ;p –  Journeyman Geek Mar 15 '10 at 3:23
4  
You say “new files”, but unless these files are fresh, random data, you are probably seeing the benefit of block-level syncing (like in rsync and other tools). –  Chris Johnsen Mar 15 '10 at 3:32
1  
This is more of a hosting comparison imo, I know FTP servers that are faster than Dropbox and I also use multiple connections with Filezilla so the statements listed in this answers do not hold. –  Tom Wijsman Jul 18 '10 at 21:00
    
Dropbox does use de-duplication to save on storage space of common files, so it does not need to upload them if it has them already. –  paradroid May 9 '11 at 15:53

6 Answers 6

There could be a number of reasons for this.
The FTP protocol is far from efficient.

  1. An FTP transfer needs at least two connections (one for control and one for data) where DropBox may be using just a single HTTP connection. Also the data connection for an FTP session may be opened from the server to your client and if you are NATed this may fail so your FTP client may be trying to connect that way around, failing then trying the other way around.

  2. There is a lot of to-ing and fro-ing on an FTP connection. To send a file the client needs to send a minimum of two commands (one to open the data connection and one to start the send) and each time it needs to wait for the server to respond, adding extra latency. As well as these two round-trips per file there are several command-response round-trips for the initial connection - one to send the username, one for the password, and at least one to set transfer parameters (to make sure the server is expecting binary, not ASCII, data). The client may also issue a couple of extra commands to get information back from the server about itself. Dropbox is likely to be using just that one HTTP request, or at most two (one to authenticate, one to send the data).

  3. On top of this, depending on what client you are using for FTP transfers (which you don't specify it would be a good idea to edit your question to include that information) it may be dropping the connection after each send operation and reconnecting next time. It is not unlikely that DropBox maintains a connection open for a while for the purposes of long-polling, to react as soon as it can to new data being available that this client should download, so it while it will need to bring up a new HTTP connection to send a file it won't need to re-authenticate.

  4. It is not unlikely that the DropBox client is compressing data before sending it (to improve speed and save bandwidth) where your FTP client won't be. So even for larger files (unless they are pre-compressed or encrypted) DropBox, and utilities like it, may be faster than a basic FTP transfer by some margin.

For large files, the first three points above will pale into insignificance compared to the time taken to actually transfer the data, but point 4 may still be quite important. For small files all the extra setup time added by the FTP protocol can potentially be a couple of times longer than the time taken to actually send the data.

share|improve this answer
    
+1 for the detailed answer. I too had wondered how Dropbox was so quick. –  Grant Palin Mar 15 '10 at 0:49
    
Really excellent, thank you very much :) –  asksuperuser Mar 15 '10 at 7:08
    
I read somewhere that the dropbox data is encrypted before transferring - so it would make sense that it's also (at least a little) compressed. –  Dean Rather Mar 29 '10 at 23:34
    
An encrypted file shouldn't be compressable - I don't drop box encrypts files during transfer anyway –  Martin Beckett Jul 19 '10 at 1:12
1  
I'm pretty sure they use HTTPS for transfer (HTTP over SSL) rather than sending data in plain form. I don't know what (if any) encryption is used for the actual storage, but if your data is sensitive you should be encrypting it at your side anyway so only you have a copy of the relevant keys. –  David Spillett Jul 19 '10 at 20:43

As others have mentioned, Dropbox can skip parts of files that have not changed. But also, Dropbox will skip uploading files if it already has a copy on the server side (one that you or anyone else has already uploaded).

So, if you are trying to upload a file that is identical to a file that Dropbox already has, the upload is skipped (and the other linked machines can start downloading it from the Dropbox servers). If you are uploading a file that is nearly identical to another, already uploaded file (it is not clear whether the already uploaded file has to be ‘yours’ or could have come from any user), then it will just send enough parts of the file to recreate it on the server when combined with the file that was already uploaded.

FTP can do none of these things (it is a simple protocol to send and receive streams of data without reference to any other data that is available on the remote end). Tools like rsync and Unison can ‘skip chunks that the other side already has’, but are usually limited to comparing chunks inside files at an identical path in the synchronized hierarchy. Dropbox appears to extend this idea to collections of files (so if you ‘upload’ two nearly identical files, presumably it could arrange to only send one plus enough of a ‘diff’ to re-create the other).

share|improve this answer

I assume you mean faster in terms of transferring files. When you save a file in your Dropbox folder, Dropbox only sends the delta (or diff) of the data to the remote storage server. FTP (most likely) sends the file byte by byte (rather than just sending the changes), which potentially takes much longer to transfer over a network. Similarly, when syncing from the remote server, the local clients will download only the changes.

The LAN sync feature can also potentially speed up syncs and reduce the network traffic needed.

share|improve this answer
    
Indeed I'm talking about new files for both cases. –  asksuperuser Mar 14 '10 at 23:30

Dropbox might be faster when you send larger quantity of files. FTP is as fast as you can get when we talk speed but it takes too much "talk" between server and client computer for each file, so the ftp seems to be slower. If you are uploading some open source application with thousands of files, it is more convenient to compress all files, upload it via FTP and decompress it on server.

share|improve this answer

I guess they use simple hashing techniques similar to md5/sha

Whenever you drop a file inside local "dropbox", dropbox-client computes hash of that file and must be sending some extra data like filesize, filename to the dropbox-server.

If dropbox-server finds similar files (they must be maintaining index of hashes and file-data on their server) it will simply inform client that file has been "uploaded" successfully. ;-)

This way you end up "uploading" file logically only. As there is no real file-content transfer, this has to be faster than anything else.

I am not sure which hashing algorithm dropbox uses, but I am 100% sure their working principle is similar to one I outlined above.

share|improve this answer

Although Dropbox is using other services, they have historically been using Amazon AWS (Amazon Web Services). It sounds like your transfer from the source to destination has a very big transfer pipe. In my experience, Dropbox is using a destination that can accept large amounts of data at once. Dropbox also distributes the upload to different IP addresses. The site you are FTPing to likely has a much smaller transfer pipe and does not have the ability to distribute uploads as efficiently.

If you run Resource Monitor (resmon), and go to the Network tab, you will notice the different processes using the network bandwidth.

  • Under Processes with network Activity, select the column for Total (B/sec)
  • Under TCP Connections, select the column for Total (B/sec)

For me, when I am uploading a file to Dropbox, it is using 4 connections to send 4 different IP addresses.

enter image description here

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.