Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing.

How can I do them simultaneously?

share|improve this question

2 Answers 2

The question is tagged with curl but the answer only uses wget.

With curl it's a little easier than wget because it can request compression and decompress without piping (url truncated for clarity).

curl --compressed http://freebase.com/topic.bz2
share|improve this answer
    
Looks like it does the trick. Great answer @user23337 –  Alec Sep 30 '13 at 15:50
up vote 2 down vote accepted

This:

wget -O - -o /dev/null http://download.freebase.com/datadumps/latest/freebase-simple-topic-dump.tsv.bz2 | bunzip2 > freebase-simple-topic-dump.tsv

Where bunzip2 is an unzipping command for your compression format of choice. It must support piped input. And the file must be a single compressed file, not an archive.

It uses wget to pipe the downloaded file to the unzipping application, outputting to the specified filename.

share|improve this answer
3  
Technically, it doesn't do them simultaneously, it just performs the download via wget first and then pipes the results as a whole into bunzip. If you attempted to unzip a file that wasn't completely written you'd get an error indicating that the end of file was reached too soon. –  MaQleod Oct 7 '12 at 4:20
    
@MaQleod Sorry, I'm pretty sure that's incorrect. Try running it without forwarding the output to a file; it starts printing straight away. –  Alec Oct 7 '12 at 4:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.