Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have downloaded wikipedia in a compressed form. It is a .7z file, which contains a .tar file. In compressed form it is only 14 GB. If I uncompress it, it has 208 GB in it. And I don't have that much space on my hard drive.

I guess there is a way to use it in its compressed form, by using some sort of mount tools.

I am on Windows Vista.

What to do?

ServerFault question reference.

share|improve this question
    
I wasn't sure which would be a better fit, so I posted at both. –  dharm0us Dec 30 '09 at 17:14
    
It may be time for an external drive. I've been seeing sales were 500GB drives were less than $50, 1TB drives less than $90. –  pcapademic Dec 30 '09 at 19:49
    
Sweet. Where did you manage to get your hands on that? –  snicker Dec 31 '09 at 0:43
    
please do NOT post the same question on multiple sites –  Jeff Atwood Dec 31 '09 at 10:28
add comment

6 Answers

I think you should extract the tar.7z file once to its whole 200GB+ size,
Then you can re-zip the base directory as a pure 7z archive.

It will probably not exceed 14GB by far.

You can then browse the new 7z archive more easily.
this will need one go of un-zip, un-tar and re-zip that will take disk space and compute time though.

share|improve this answer
add comment

Any decent Orhtodox File Manager allows you to browse through archives. I know Total Commander has 7zip plugin that will allow you to browse the 7z archive as a directory.

share|improve this answer
    
Tried it. Doesn't help. It again tries to extract the whole 208 GB. –  dharm0us Dec 30 '09 at 17:15
    
I know this might sound obvious, but does browsing the archive with 7z itself causes the same issue? It doesnt look like it actually extracts the archive that way. –  Vitaliy Dec 30 '09 at 17:22
    
Yes, browsing the archive with 7zip causes the same behavior. –  dharm0us Dec 30 '09 at 17:27
    
Gnome Commander might do the trick –  blueberryfields Dec 30 '09 at 18:12
add comment

On a Unix box (or with cygwin (Unix for windows), a Unix environment), you could pipe the file, and get a listing out of it, or probably make a zip file of the whole thing, which would allow random access.

In some ways, tar is a feature, as it was meant for tape drives (one way linear access).

Get a Linux vm and a virtual env (say, vmware server?), or cygwin, and have fun.

7z -options_to_extract_to_console/pipe wiki9.tar.7z | tar t > listing_of_files.txt

would give you a list of what was there

Then

7z -same_options_as_before wiki9.tar.7z | tar -xT list_of_files_I_want

and it should write them (possibly someplace weird).

I think. I don't have the listing for 7z switches on Unix handy right now.

Or to make it a zip archive:

7z -same_options_as_before wiki9.tar.7z | tar -x | zip wiki.zip -

Note that it will likely be 2-4 times larger (better to stick the files together and then compress than compress one at a time, and 7z uses better compression).

Expect it to take hours.

Consider borrowing an external drive for a few days :-D or upgrade your HD, heck under $100 for internal or external over 200gb, even laptop HD.

share|improve this answer
    
$100 for about 1/2 TB these days. –  Chris Oct 11 '10 at 17:55
add comment

Open it with 7Zip? I will only decompress what you tell him to...might take some time, though.

Edit: Wait a minute, I just realized that this is possibly a dump idea because of the size. I don't know how 7Zip exactly works, but I'd guess that he's just extracting the file list, without caching the whole content.

share|improve this answer
add comment

You basically cannot accomplish this. 7zip's compression style doesn't lend itself to opening random sections without opening the whole file. If I recall correctly, it's a stream system.

For example, this is different than with .zip archives which will compress each file individually (with whatever algorithm they fancy), and .bz2 will use a fixed (typically) 900kb block size. Both of those forms can be used to seek to an individual space without full compression.

And, just to add insult in case I'm wrong about the .7zip story: Tar files don't store their file information in a central location, so you have to expand that too. See http://en.wikipedia.org/wiki/Tar%5F%28file%5Fformat%29#Random%5Faccess

You'll essentially have to find the disk space if you want to use those files.

share|improve this answer
    
Not true if you think like a Unix user, you do have to uncompress the whole thing, but you can pipe it to grab only what you want, and not use any space other than for what you save. –  Ronald Pottol Dec 31 '09 at 0:37
add comment

FYI, the Wikipedia data dumps are typically just one XML file.

A few years ago I wrote a Wikipedia data dump parser in C: wikiq. It converts data dump streams into a tab-separated revision history format amenable to use in statistical processing programs.

7za e -so <file> | ./wikiq | <tab-separated processing program>

If you throw away text and comment data (as it does by default), processing is actually bounded by decompression (20-30mb/s).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.