37

I'd like to know why technically Dropbox is much faster than FTP? What kind of technology does it use?

I'm not talking about diff files, I'm talking about transferring new files in both cases, Dropbox is much faster.

I mean it, very much faster, maybe 10 times faster than FTP for files I uploaded. I will experiment again for bigger files later.

5
  • 2
    What size, type, and number of files did you upload? How long did each of them take to upload? Where were you uploading the files to via FTP? Dropbox is not magic, the simplest explanation is that the FTP server you were uploading too has much less bandwidth than Amazon does.
    – user23307
    Mar 15, 2010 at 1:50
  • 2
    if they already have it, it dosen't re-upload ;p
    – Journeyman Geek
    Mar 15, 2010 at 3:23
  • 4
    You say “new files”, but unless these files are fresh, random data, you are probably seeing the benefit of block-level syncing (like in rsync and other tools). Mar 15, 2010 at 3:32
  • 1
    This is more of a hosting comparison imo, I know FTP servers that are faster than Dropbox and I also use multiple connections with Filezilla so the statements listed in this answers do not hold. Jul 18, 2010 at 21:00
  • Dropbox does use de-duplication to save on storage space of common files, so it does not need to upload them if it has them already.
    – paradroid
    May 9, 2011 at 15:53

6 Answers 6

30

There could be a number of reasons for this.
The FTP protocol is far from efficient.

  1. An FTP transfer needs at least two connections (one for control and one for data) where DropBox may be using just a single HTTP connection. Also the data connection for an FTP session may be opened from the server to your client and if you are NATed this may fail so your FTP client may be trying to connect that way around, failing then trying the other way around.

  2. There is a lot of to-ing and fro-ing on an FTP connection. To send a file the client needs to send a minimum of two commands (one to open the data connection and one to start the send) and each time it needs to wait for the server to respond, adding extra latency. As well as these two round-trips per file there are several command-response round-trips for the initial connection - one to send the username, one for the password, and at least one to set transfer parameters (to make sure the server is expecting binary, not ASCII, data). The client may also issue a couple of extra commands to get information back from the server about itself. Dropbox is likely to be using just that one HTTP request, or at most two (one to authenticate, one to send the data).

  3. On top of this, depending on what client you are using for FTP transfers (which you don't specify it would be a good idea to edit your question to include that information) it may be dropping the connection after each send operation and reconnecting next time. It is not unlikely that DropBox maintains a connection open for a while for the purposes of long-polling, to react as soon as it can to new data being available that this client should download, so it while it will need to bring up a new HTTP connection to send a file it won't need to re-authenticate.

  4. It is not unlikely that the DropBox client is compressing data before sending it (to improve speed and save bandwidth) where your FTP client won't be. So even for larger files (unless they are pre-compressed or encrypted) DropBox, and utilities like it, may be faster than a basic FTP transfer by some margin.

For large files, the first three points above will pale into insignificance compared to the time taken to actually transfer the data, but point 4 may still be quite important. For small files all the extra setup time added by the FTP protocol can potentially be a couple of times longer than the time taken to actually send the data.

9
  • +1 for the detailed answer. I too had wondered how Dropbox was so quick. Mar 15, 2010 at 0:49
  • 1
    I read somewhere that the dropbox data is encrypted before transferring - so it would make sense that it's also (at least a little) compressed. Mar 29, 2010 at 23:34
  • 1
    An encrypted file shouldn't be compressable - I don't drop box encrypts files during transfer anyway Jul 19, 2010 at 1:12
  • @mgb: you are correct that file compression techniques should not find enough redundancy in encrypted to data to be useful, so initially sending a file will result in no help from compression. But if dropbox already has the file and you have just updated it (and the key is still the same) then chances are it will not need to transfer the entire file to update the remote copy. While the data can not be compressed, the amount you need to send to keep it uptodate can still be reduced (considerably for large files that see small updates). Jul 19, 2010 at 12:58
  • 1
    I'm pretty sure they use HTTPS for transfer (HTTP over SSL) rather than sending data in plain form. I don't know what (if any) encryption is used for the actual storage, but if your data is sensitive you should be encrypting it at your side anyway so only you have a copy of the relevant keys. Jul 19, 2010 at 20:43
14

As others have mentioned, Dropbox can skip parts of files that have not changed. But also, Dropbox will skip uploading files if it already has a copy on the server side (one that you or anyone else has already uploaded).

So, if you are trying to upload a file that is identical to a file that Dropbox already has, the upload is skipped (and the other linked machines can start downloading it from the Dropbox servers). If you are uploading a file that is nearly identical to another, already uploaded file (it is not clear whether the already uploaded file has to be ‘yours’ or could have come from any user), then it will just send enough parts of the file to recreate it on the server when combined with the file that was already uploaded.

FTP can do none of these things (it is a simple protocol to send and receive streams of data without reference to any other data that is available on the remote end). Tools like rsync and Unison can ‘skip chunks that the other side already has’, but are usually limited to comparing chunks inside files at an identical path in the synchronized hierarchy. Dropbox appears to extend this idea to collections of files (so if you ‘upload’ two nearly identical files, presumably it could arrange to only send one plus enough of a ‘diff’ to re-create the other).

10

I assume you mean faster in terms of transferring files. When you save a file in your Dropbox folder, Dropbox only sends the delta (or diff) of the data to the remote storage server. FTP (most likely) sends the file byte by byte (rather than just sending the changes), which potentially takes much longer to transfer over a network. Similarly, when syncing from the remote server, the local clients will download only the changes.

The LAN sync feature can also potentially speed up syncs and reduce the network traffic needed.

1
  • Indeed I'm talking about new files for both cases.
    – asksuperuser
    Mar 14, 2010 at 23:30
1

Although Dropbox is using other services, they have historically been using Amazon AWS (Amazon Web Services). It sounds like your transfer from the source to destination has a very big transfer pipe. In my experience, Dropbox is using a destination that can accept large amounts of data at once. Dropbox also distributes the upload to different IP addresses. The site you are FTPing to likely has a much smaller transfer pipe and does not have the ability to distribute uploads as efficiently.

If you run Resource Monitor (resmon), and go to the Network tab, you will notice the different processes using the network bandwidth.

  • Under Processes with network Activity, select the column for Total (B/sec)
  • Under TCP Connections, select the column for Total (B/sec)

For me, when I am uploading a file to Dropbox, it is using 4 connections to send 4 different IP addresses.

enter image description here

0

Dropbox might be faster when you send larger quantity of files. FTP is as fast as you can get when we talk speed but it takes too much "talk" between server and client computer for each file, so the ftp seems to be slower. If you are uploading some open source application with thousands of files, it is more convenient to compress all files, upload it via FTP and decompress it on server.

0

I guess they use simple hashing techniques similar to md5/sha

Whenever you drop a file inside local "dropbox", dropbox-client computes hash of that file and must be sending some extra data like filesize, filename to the dropbox-server.

If dropbox-server finds similar files (they must be maintaining index of hashes and file-data on their server) it will simply inform client that file has been "uploaded" successfully. ;-)

This way you end up "uploading" file logically only. As there is no real file-content transfer, this has to be faster than anything else.

I am not sure which hashing algorithm dropbox uses, but I am 100% sure their working principle is similar to one I outlined above.

You must log in to answer this question.