Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm downloading a large file that's not an archive, and I want to combine the tasks of downloading and decompressing.

How can I do them simultaneously?

share|improve this question

The question is tagged with curl but the answer only uses wget.

With curl it's a little easier than wget because it can request compression and decompress without piping (url truncated for clarity).

curl --compressed http://freebase.com/topic.bz2
share|improve this answer
    
Looks like it does the trick. Great answer @user23337 – Alec Sep 30 '13 at 15:50
up vote 2 down vote accepted

This:

wget -O - -o /dev/null http://download.freebase.com/datadumps/latest/freebase-simple-topic-dump.tsv.bz2 | bunzip2 > freebase-simple-topic-dump.tsv

Where bunzip2 is an unzipping command for your compression format of choice. It must support piped input. And the file must be a single compressed file, not an archive.

It uses wget to pipe the downloaded file to the unzipping application, outputting to the specified filename.

share|improve this answer
3  
Technically, it doesn't do them simultaneously, it just performs the download via wget first and then pipes the results as a whole into bunzip. If you attempted to unzip a file that wasn't completely written you'd get an error indicating that the end of file was reached too soon. – MaQleod Oct 7 '12 at 4:20
    
@MaQleod Sorry, I'm pretty sure that's incorrect. Try running it without forwarding the output to a file; it starts printing straight away. – Alec Oct 7 '12 at 4:32
1  
I agree with @Alec as it is also faster – Karussell Nov 3 '15 at 16:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .