Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

If I have a text file in Unix that contains N-many independent entries (e.g. records about employees, where each employee has a separate record), is it expected that this file will take up less space than if I split the file into N files, each containing the entry for one employee? in other words, can one save significant space on unix file systems by concatenating many files together, or is the difference negligible? thanks.

share|improve this question

1 Answer 1

It depends on the relationship between the record lengths and filesystem block size.

For example, if the record lengths (modulo the block size) are randomly distributed, concatenating the data into fewer files will reduce wasted space. The last block in each file will be 1/2 full on average. This is what's called internal fragmentation. If you cut the number of files in half by combining files, you would reduce the number of 1/2 full blocks in half.

Otoh, if every record is always a multiple of the block size, every file will always completely fill every block allocated to it (zero internal fragmentation) no matter how many records it contains. In that case, the only savings from concatenating files would be whatever came from having fewer directory entries.

share|improve this answer
    
On the other other hand, if the records are short, each one will get a full disk block, wasting much space. –  vonbrand Jan 23 '13 at 19:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.