I have 4 text files, each with about 17 million rows (or lines, if you will). The files are named 1.txt, 2.txt, 3.txt and 4.txt. Text file 1.txt contains the following sample data;


Just a pair of numbers separated by a comma and a colon at the end. Text file 2.txt contains the following sample data;


Just groups of three numbers separated by commas with the opening and closing brackets at the begining and end. Text file 3.txt contains the following sample data;


Just 6-character hexadecimal numbers with a pound sign at the start. Lastly, text file 4.txt contains the following sample data;


What I've been trying to do is combine all the four text files into one, separated by tabs. Output should be like this;

0,0:    (0,0,0)     #000000     srgb(0,0,0)
1,0:    (0,0,257)   #000001     srgb(0,0,1)
2,0:    (0,0,514)   #000002     srgb(0,0,2)
3,0:    (0,0,771)   #000003     srgb(0,0,3)

I have tried

paste -d "\t" 1.txt 2.txt 3.txt 4.txt> final.txt

but i get some weird result, a sample of which looks like

0,0:    (0,0,0)     #000000
1,0:    (0,0,257)   #000001
2,0:    (0,0,514)   #000002
3,0:    (0,0,771)   #000003

The problem is that the fourth column jumps over a new line which is unexpected for me. Any solutions to fix this? I am on Windows 8.1 and I have Git installed to run Linux commands.

  • @KamilMaciorowski Yes 3.txt has no commas whatsoever so I guess it's different than the others. Is that a problem? As to how I examined final.txt, I have the EmEditor text file reader installed, which is able to open really large text files – Kara Kirkland Jul 31 '19 at 21:34
  • @KamilMaciorowski, I forgot to mention that the lines in 3.txt don't end in any special character – Kara Kirkland Jul 31 '19 at 21:46

My guess is 3.txt (and maybe 4.txt) uses CR+LF line endings (DOS/Windows style), the other files use sole LF (Unix style). paste expects lines ended with LF, it treats CR as a regular character. In effect you got CR in the middle of each expected line of the output.

Some text editors are flexible, they go to the next line after detecting CR+LF, sole CR or sole LF. This way the excessive CR characters generate additional lines when you examine the file: in the sample instead of four lines you see eight. Many Linux tools will still perceive four lines there.

In Linux file *.txt would inform you about foreign line endings; dos2unix 3.txt would fix them.

The reason one of the files contains CR+LF may be you generated it with some Windows tool, the file came from the Windows world.

Also note POSIX requires all lines to end with LF, while in Windows (I think) the last line may not end with CR+LF. I know dos2unix doesn't add the missing LF at the very end (other converters may). Incomplete last line may make Linux tools complain or "misbehave" (e.g. ignore the line). Opening the file in a Linux text editor and saving may fix it; in general this depends on the editor and its configuration though.

  • Indeed, 3.txt was generated by a windows tool and all the files are from a windows environment. Running dos2unix on all the text files and then running the desired paste command surely fixed the issue. – Kara Kirkland Aug 1 '19 at 12:52
  • Oh, and I didn't have any issue at all with the last line(s). – Kara Kirkland Aug 1 '19 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.