Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am going to sort a REALLY BIG delimited text file, say 250Mb (or a bunch of files of more or less than 250Mb) . It have 37 fields, and I need to sort it by 5 fields, for example 1st, 4th, 5th, 6th 7th fields. Under Unix / VMS, do I have a good option to do this FAST? I can write COBOL program.
Now I am trying to sort them using the below command, but it already run for a long time and just not going to finished.

Thank you.

The command I used: time sort -t ',' -o sorted.txt +0 -1 +4 -5 +5 -6 +6 -7 +22 -23 *.DAT_gprscdr_ftpd

share|improve this question

migrated from programmers.stackexchange.com Feb 9 '11 at 12:51

This question came from our site for professional programmers interested in conceptual questions about software development.

2  
Ask this question at stackoverflow.com –  Amir Rezaei Feb 9 '11 at 7:50
1  
Look at this SO question stackoverflow.com/questions/930044/… you might be able to use the script in there. –  KeesDijk Feb 9 '11 at 8:03
1  
What's wrong with the shell script you provided? The unix sort is fast. –  S.Lott Feb 9 '11 at 11:20
add comment

2 Answers

up vote 3 down vote accepted

Maybe this question really should be in another SE-site, but here is my take on this issue.

1) Isn't the basic sort you provided in your question fast enough? How fast it should be? My 2 year old desktop sorts 270MB of Apache access log files in 21 seconds.

2) If that is not fast enough, you can try to first sorting each file individually and then merge them with "sort -m"

3) If not fast enough and you have more than one CPU/core, parallelize (sp?) the process with GNU Parallel

4) If still not fast enough and you have more machines available, parallelize the sorting process on multilple machines with GNU Parallel

share|improve this answer
add comment

you can load the data in MySQL database (LOAD command) and do what you want

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.