Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to combine two or more files in Linux, so I am using the following command:

cat small_file LARGE_File LARGER_FILE > SUM_OF_FILES

However this runs very slow.

Does anyone know a Linux tool that combines the files in the fastest time?

share|improve this question

migrated from stackoverflow.com Nov 22 '11 at 22:14

This question came from our site for professional and enthusiast programmers.

4  
Get yourself faster hard drives. The bottleneck is not in the command, but in the speed of reading the data. – Dark Falcon Nov 22 '11 at 17:14
    
How often do you need to do this? If frequently then consider a different system of running things. – Ed Heal Nov 22 '11 at 17:20
    
@Ed Heal: periodically – macki Nov 22 '11 at 17:23
2  
Use a different physic hard drive for the result file. – Codism Nov 22 '11 at 17:49
    
@macki: "periodically" could mean 10 times per second or once every 10 years. – Keith Thompson Nov 22 '11 at 18:51
up vote 5 down vote accepted

You could try a variation on the dd command, such as:

dd if=small_file bs=4k of=SUM_OF_FILES

dd if=LARGE_FILE bs=4k of=SUM_OF_FILES oflag=append

dd if=LARGER_FILE bs=4k of=SUM_OF_FILES oflag=append
share|improve this answer
    
this is what I'm trying right now, I'm having problem with the exact size – macki Nov 22 '11 at 22:10

I've found mmv (Mass Move and rename - Move, copy, append or link Multiple files using wildcard patterns.) from this useful bash reference. So you could do:

cp small_file SUM_OF_FILES
mmv -a LARGE_File SUM_OF_FILES
mmv -a LARGER_FILE SUM_OF_FILES

(note: mmv isn't installed by default, use sudo apt-get install mmv)

share|improve this answer
    
brew install mmv to install mmv on max osx. – suzanshakya Dec 21 '14 at 6:01

Maybe

cat small_file >> LARGE_File

will do what you want? If you need LARGE_FILE to be unchanged

cp LARGE_File SUM_OF_FILES
cat small_file >> SUM_OF_FILES

is better, but this will only be slightly faster than your original code.

share|improve this answer
2  
this is very slow when merging LARGE FILES – macki Nov 29 '11 at 18:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .