Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to (recursively) download a directory of files to a location from a WebDav Server. If the file is alreay present (somewhere there), it shall not be downloaded again. However the folder structure is not the same.

Is there an easy way to do that? I looked into fdupes, but it's just for detection and deletion of dupes. The files are very large and the overhead would be by far too great.

The target filesystem doesn't support deduplication. I know cp -n (from a FUSE mountpoint) wouldn't overwrite the existing files, but the folder structure isn't the same. So I'm kind of stuck.

share|improve this question

1 Answer 1

up vote 3 down vote accepted

From looking at the available Linux clients for WebDAV, my own preferred method of doing this would be:

  1. Use GVFS or one of the WebDAV filesystem modules (davfs2 or fusedav) to "map" the remote WebDAV server's files into the local filesystem path.

  2. Use the built-in cp command to with the -n option to instruct it to "not clobber" files in the destination. Note that certain shells, such as dash on Ubuntu, will execute a builtin version of cp by default, and this builtin might not support the -n option. For best results, make sure you are executing the GNU Coreutils version of cp by running /bin/cp or /usr/bin/cp (depending on where the binary is on your specific system).

EDIT: I misread your original question.

I think what you're saying is that you have the situation where file file1.txt exists in two different paths in the WebDAV server, and the contents of those two files is exactly the same. And since you already have one copy of the file, you don't want to download a second or third copy of the file because it wastes bandwidth?

Well, from the client side, this would be very hard to do. Here's why.

You have to look at what you are comparing to determine if the file is unique, and the requirements/costs to make that comparison.

I assumed (wrongly) that what you were comparing is the path relative to the root of the WebDAV folder structure. The cost of making a path equality comparison is very easy: you just look at the two path strings, like /dir1/dir2/file1.txt, and see if the strings match. If they do, it's a duplicate. If they don't, it's not.

Another thing you could compare is the file name, ignoring the path. So, for example, would you consider these two files duplicate: /dir1/dir2/file1.txt and /dir3/dir4/file1.txt? Well, if you're only comparing based on the name, then these would be considered duplicates. However, we can mix and match various tests for duplication as we like, in order to make the right kind of test for our use case.

Other, less-useful properties to compare include file size, attributes (also known as metadata), file extension, etc. These things do not give you very much to go on in terms of duplicate-judging power, because in most cases, it is easy to construct a file which has the same properties as another file but entirely different contents, and most people wouldn't consider the two files to be duplicates if the contents differ.

In my opinion, the most important thing you can compare is the file contents. Unfortunately, from a WebDAV client's perspective, you have no way of knowing the file contents until you've already downloaded the file. And as far as the client is concerned, the file contents could change during or after the file transfer, in which case, the results of the duplicate comparison would change if you re-downloaded the file.

There are two basic ways to compare file contents: byte for byte, and hashing. Byte for byte is the most "guaranteed" way to check for duplicates, but it suffers from the limitation that you have to compare the entire file, which is enormously slow for a large amount of data. Also consider that the basic algorithmic complexity of duplicate detection is O(n^2), which means that you would have to compare each file's contents against each other file's contents in order to determine if it's a duplicate. Using a cryptographic hash to compare the files can vastly reduce the amount of data that has to be compared or transferred, but the downside is that you introduce an infinitesimally small chance that two files can be actually different but have the same hash -- known as a hash collision.

But again, from the client perspective, it is not possible to know what the file contents are, or even its hash, unless you either:

  • Download the file from the server; or
  • Convince the server to compute a hash value for you locally, and then download the hash.

In the former case, you are downloading the file to determine if it's a duplicate to avoid downloading the file, so you can't do that, obviously -- you're wasting the bandwidth you're trying to avoid just to perform the comparisons!

In the latter case, you might be on to something. A SHA1 hash of a very large file is only a couple of bytes, and represents a tiny fraction of the size of the large file. It would be fairly practical to download hashes of all the files and do an O(n^2) comparison of the hashes to determine which file to download. You still run into race condition issues if the file data changes on the server while you're doing these comparisons, though, so you need to make sure you take synchronization into account if it's important to you.

So, conclusion:

  • IF you do not have full software control over the WebDAV server and are unable to make changes to its configuration, you are pretty much Out of Luck (tm) in determining whether you already have a copy of the same file contents which are stored in multiple files on the server, unless the server admin already makes available some kind of hash files for every file on the server, which might allow you some degree of success if you can rely on the hash values.
  • IF you do have full software control over the WebDAV server and are able to change its configuration, you may want to write a script or a program (or use one already available) to create a hash file with an extension such as, e.g. .sha1sum in the same directory as every file hosted by the WebDAV server. This could allow you to download only the hashes and compare them, at a relatively modest bandwidth cost compared to the size of the files, assuming your files are more than a couple kilobytes in size.
share|improve this answer
    
I mean that the file is not present in any subfolder, or anywhere else in that location –  wishi Jan 3 '13 at 18:17
1  
OH. You mean the file exists in path /dir1/dir2/file1.txt AND /dir3/dir4/file1.txt and the two files are copies of eachother? This would be a problem. :S –  allquixotic Jan 3 '13 at 18:59
    
Thank you for that answer. The approach with hashsums and a list is the way to go. With some Python. I didn't know whether there is some well known out-of-the box trick with some awk/sed/cp/md5sum magic :) Sometimes there is. And usually I'm the last person who knows. –  wishi Jan 4 '13 at 9:47
1  
Hey, if you manage to develop something in Python that's generally useful, you should publish your code on github, etc. and edit my answer (and/or your question) providing what you learned :) That'd be immensely helpful to others who have the same problem. –  allquixotic Jan 4 '13 at 16:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.