Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to combine two or more files in Linux, so I am using the following command:

cat small_file LARGE_File LARGER_FILE > SUM_OF_FILES

However this runs very slow.

Does anyone know a Linux tool that combines the files in the fastest time?

share|improve this question

migrated from stackoverflow.com Nov 22 '11 at 22:14

This question came from our site for professional and enthusiast programmers.

4  
Get yourself faster hard drives. The bottleneck is not in the command, but in the speed of reading the data. –  Dark Falcon Nov 22 '11 at 17:14
    
How often do you need to do this? If frequently then consider a different system of running things. –  Ed Heal Nov 22 '11 at 17:20
    
@Ed Heal: periodically –  macki Nov 22 '11 at 17:23
2  
Use a different physic hard drive for the result file. –  Codism Nov 22 '11 at 17:49
    
@macki: "periodically" could mean 10 times per second or once every 10 years. –  Keith Thompson Nov 22 '11 at 18:51

3 Answers 3

up vote 4 down vote accepted

You could try a variation on the dd command, such as:

dd if=small_file bs=4k of=SUM_OF_FILES

dd if=LARGE_FILE bs=4k of=SUM_OF_FILES oflag=append

dd if=LARGER_FILE bs=4k of=SUM_OF_FILES oflag=append
share|improve this answer
    
this is what I'm trying right now, I'm having problem with the exact size –  macki Nov 22 '11 at 22:10

I've found mmv (Mass Move and rename - Move, copy, append or link Multiple files using wildcard patterns.) from this useful bash reference. So you could do:

cp small_file SUM_OF_FILES
mmv -a LARGE_File SUM_OF_FILES
mmv -a LARGER_FILE SUM_OF_FILES

(note: mmv isn't installed by default, use sudo apt-get install mmv)

share|improve this answer
    
brew install mmv to install mmv on max osx. –  suzanshakya Dec 21 at 6:01

Maybe

cat small_file >> LARGE_File

will do what you want? If you need LARGE_FILE to be unchanged

cp LARGE_File SUM_OF_FILES
cat small_file >> SUM_OF_FILES

is better, but this will only be slightly faster than your original code.

share|improve this answer
2  
this is very slow when merging LARGE FILES –  macki Nov 29 '11 at 18:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.