Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a bunch of text logfiles in the following format:

ID          (17 characters)
Timestamp   (14 characters YYYYmmddHHMMSS e.g. "20060210100040" -> 2006/02/10 10:00:40)
Random data (? characters)
end of line

The files are already sorted by timestamp. I need to get 1 log file with all the logs from multiple logs files, sorted by timestamp. Note that the log files are really huge, around 3-4G each (and there are dozens of them) I tried the following command:

sort -s -m -t '|' -k1n,1n +17 -o data_sort.txt *.TXT

Here is how I ended up with this command:

-s     : don't bother with tie results
-m     : merge all logs files
-t '|' : there is no | in my logs, so the whole line should be field 1
-k1n,1n: sort on the first field as a numeric value
+17    : the timestamp starts at index 17
-o     : output file

Actually... it fails miserably. The output file data_sort.txt is just the concatenation of all files, not sorted at all :(

I would greatly appreciate if anyone could provide any help on this problem!

Thanks

share|improve this question
    
Any answer that suggests concatenating the files first clearly hasn't taken account of the fact that their sizes are given in GB. –  OrangeDog Apr 17 '13 at 16:45

4 Answers 4

up vote 2 down vote accepted

Your key should be -k1.17n and omit the -t and the +17.

Is there a space between the ID and the timestamp? Then the timestamp is field 2 and the key should be -k2.

share|improve this answer
    
The problem is that there may be a space in the ID, so the timestamp could be field 1 or field 2, that is why i tried to fix it to field 1 with the -t '|' trick :/ –  NewbiZ Jun 3 '10 at 15:43
    
@NewbiZ: Then -k1.17n should work. –  Dennis Williamson Jun 3 '10 at 16:23
    
It worked! thanks –  NewbiZ Jun 3 '10 at 16:47

man sort reads:

-m, --merge merge already sorted files; do not sort

The '+' symbol doesn't show up in my man page for sort. So I don't know how you get +17. If you want to use the whole line, you do not need -t or -k, since the default is to start sorting from beginning of line, to end of line.

share|improve this answer
    
-m is a bit ambiguous info coreutils sort states: Merge the given files by sorting them as a group. Each input file must always be individually sorted. A thought it would merge&sort locally sorted files. '+' does not show up in my man too, but i've seen people using in on some web pages, and i did not find any other way to specify where is the timestamp in the line –  NewbiZ Jun 3 '10 at 14:10

I like these hard ones...this one got me thinking:

Essentially, it concatenates all the .txt files, separates them with colons (for sorting), sorts the second field (the r sorts newest first, take it out if you want newest last), and then removes the colons, showing the original line.

cat *.txt
 | awk '{print substr($0,1,17)":"substr($0,18,14)":"substr($0,32)}'
 | sort -t: -k2,2 -nr -s
 | tr -d ':'

I've tested it with three 4-line .txt files.

First File

1234567890123456720100603104500Random text or data
2345678901234567820100602104500New Random Text
3456789012345678920100509213849Earlier Date
4567890123456789020100521195058InBetween Date

Second File

1234567890123456720100603124500File2 Random text or data
2345678901234567820100602124500File2 New Random Text
3456789012345678920100519213849File2 Earlier Date
4567890123456789020100523195058File2 InBetween Date

Third File

12345678901234567201106031045003Random text or data
23456789012345678201004021045003New Random Text
34567890123456789201007092138493Earlier Date
45678901234567890201005231950583InBetween Date

Results

12345678901234567201106031045003Random text or data
34567890123456789201007092138493Earlier Date
1234567890123456720100603124500File2 Random text or data
1234567890123456720100603104500Random text or data
2345678901234567820100602124500File2 New Random Text
2345678901234567820100602104500New Random Text
4567890123456789020100523195058File2 InBetween Date
45678901234567890201005231950583InBetween Date
4567890123456789020100521195058InBetween Date
3456789012345678920100519213849File2 Earlier Date
3456789012345678920100509213849Earlier Date
23456789012345678201004021045003New Random Text
share|improve this answer

Try using cat first to concatenate the files and then sort that. sort won't be confused about multiple files because it will see a single input stream coming from stdin.

The combination of options you're using for -t and -k seem like you're trying to do this outside of what sort usually does. Sort operates on fields with specific delimiters -- whitespace by default.

You'll probably want to use some combination of cut (to break out fields by byte), awk to splice them together, sort to sort the lines and then awk to recreate the lines in their original format.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.