Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is there a file format that handles the following use case...

I'd like to create a tar file (or whatever - I'm just using tar here b/c it's a well known file format for containing multiple files) that would be usable even if I only had access to specific chunks of said file.

For example, say I tar up my mp3 and photo collection into a 100GB tar file, then put the file into some long term storage somewhere. Later, I want to access a specific mp3 file. I don't want to download the entire 100GB tar file just to get to one mp3. In fact, let's say I can't download the entire 100GB tar file. Instead, I'd like to say "give me megabytes 10 through 19 of the 100GB tar file" and then have the mp3 magically extracted from those 10 megabytes.

Does a file format like this exist?

share|improve this question
2  
    
@Khaleel: No, the two questions have only a superficial resemblance. –  Scott Nov 29 '12 at 23:05
add comment

2 Answers

Yes, you could do that with a tar archive but to make it work, you'd need to save a log of what you wrote, telling the offsets to the various files in the archive. Otherwise, the only way to know what's in a tar archive is to read the whole thing start to finish, which defeats the idea of only downloading the part you need.

Also, if you're only able to download the archive in whole blocks, not just any arbitrary number of bytes on any boundary, it would be helpful to have a tar that can start and end reading at specified offsets within the block. My own tar does that but that's an unusual feature; lacking that, the alternative would be to head and tail the block to extract just the part you want to un-tar.

share|improve this answer
add comment

I know this is late, not sure what the OP ended up going with, but here are some suggestions:

You could create a new partition using a filesystem of your choice and copy everything you want to store to that partition. Then shrink the partition as small as you can to save space. Then you can use e.g. dd to create an image of the partition, say it's /dev/sdc3, at its simplest:

dd if=/dev/sdc3 of=backup_image.bin

For Windows Google for "dd for windows" or find an equivalent program that can make a byte-for-byte image of a partition or device.

Then, that image file can be placed in long term storage. As long as you have access to the file and your long term storage media supports random seek, you can just mount the file directly, e.g.:

sudo mkdir /media/backup
sudo mount /path/to/backup_image.bin /media/backup

For Windows, Google for "windows mount partition image".

You'll have full access to the data and only the minimum amount of reads necessary to support navigation through the filesystem and access to that file will be performed.

Another, more obvious option, is to simply not combine your files into a single archive file. Create a normal filesystem on your long term storage and just copy the files to it. Filesystems already provide the means to access specific files without accessing unneeded data, you may not have to reinvent a wheel.

Also there are proprietary formats like the one you describe, for example, backup software like Acronis can back up data to network mounted drives or wherever, into ".TIB" files that can be browsed (and data extracted) without requiring a read of the entire file. I don't know if Acronis supports browsing of Linux/OSX filesystem backups but I do know it supports this type of browsing for FAT32 and NTFS backups. I'm sure there are others.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.