1

I would like to know how to wget files from a url which are stored with bz2 compression. However, I would like to only wget the new and updated files, relative to my destination, where the files are stored uncompressed.

Typically I would use the following code:

wget http://sourcedirectory/ --mirror

As you can imagine, since the source files are compressed, and the destination files are uncompressed, this approach downloads the entire set of files instead of just the new and updated files.

Improve this question
4
  • A similar but different question can be seen here stackoverflow.com/questions/4944295/wget-skip-if-files-exist I am adding an additional level of complexity, where skip if files exist, even if they have different compression state.
    – bsuttonq
    Commented Mar 25, 2015 at 19:44
  • My current guess is that I'll need to create a list of files at the source, another list of files at the destination, and amend those lists to look for differences while ignoring compression states. I don't really know how to do that, though, except in theory.
    – bsuttonq
    Commented Mar 25, 2015 at 19:48
  • Do the archives contain a single file each or many? If they're single files, you could compare the timestamps of the remote foo.gz and the local foo.
    – terdon
    Commented Mar 26, 2015 at 1:17
  • Each archive will contain only a single file. Thanks for the suggestion, I'll explore comparing foo.gz and local foo.
    – bsuttonq
    Commented Apr 25, 2015 at 0:43

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .