Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm currently using tar for archiving some files. Problem is: archives are pretty big, contains many data and tar is very slow when listing and extracting.

I often need to extract single files or folders from the archive, but I don't currently have an external index of files.

So, is there an alternative for Linux, allowing me to build uncompressed archive files, preserving the file attributes AND having fast access list table?

I'm talking about archives of 10 to 100 GB, and it's pretty impractical to wait several minutes to access a single file.

Anyway, any trick to solve this problem is welcome (but single archives are non-optional, so no rsync or similar).

Thanks in advance!

EDIT: I'm not compressing archives, and using tar I think they are too slow. To be precise about "slow", I'd like that:

  • listing archive content should take time linear in files count inside the archive, but with very little constant (e.g. if a list of all the files is included at the head of the archive, it could be very fast).
  • extraction of a target file/directory should (filesystem premitting) take time linear with the target size (e.g. if I'm extracting a 2MB PDF file in a 40GB directory, I'd really like it to take less than few minutes... If not seconds).

Of course, this is just my idea and not a requirement. I guess such performances could be achievable if the archive contained an index of all the files with respective offset and such index is well organized (e.g. tree structure).

share|improve this question
    
Is it a requirement that the files be stored in a single file like tar or can they all reside in a directory? –  UtahJarhead Oct 5 '12 at 22:22
    
Yes, it is a requirement. EDIT: of course it's not a requirement to use tar... It could be anything. –  AkiRoss Oct 5 '12 at 22:25
    
Are you compressing tar files using gzip or bzip2, or are uncompressed tar files too slow as well? –  Daniel Beck Oct 5 '12 at 22:26
    
@Daniel, I'll add some details about this in the question. –  AkiRoss Oct 5 '12 at 22:34
    
lzo seems to be faster imo –  kobaltz Oct 5 '12 at 22:35
add comment

2 Answers 2

up vote 1 down vote accepted

I found a similar topic on serverfault.

http://serverfault.com/questions/59795/is-there-a-smarter-tar-or-cpio-out-there-for-efficiently-retrieving-a-file-store

I'm looking at DAR, which seems to be what I need, but I'll leave this question open for other suggestions.

share|improve this answer
    
Ok, I tried both DAR and XAR. XAR seems that just can't handle archiving many files (10GB test) and seems to work with tons of RAM, while DAR works pretty well and I successfully created a 30GB archive with compression and very fast index. –  AkiRoss Oct 6 '12 at 13:05
1  
DAR was definitely what I was looking for. It's a very nice tool! –  AkiRoss Oct 26 '12 at 17:43
add comment

If tar is not a requirement, a quick search says ar will allow for an indexed archive.

share|improve this answer
    
Yes, nice, but I also read ar is a bit strict with file names length. Anyway I could try performance comparison and see if it fits. –  AkiRoss Oct 5 '12 at 22:41
    
Beats me on restrictions like that. –  UtahJarhead Oct 6 '12 at 0:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.