Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am going to sort a REALLY BIG delimited text file, say 250Mb (or a bunch of files of more or less than 250Mb) . It have 37 fields, and I need to sort it by 5 fields, for example 1st, 4th, 5th, 6th 7th fields. Under Unix / VMS, do I have a good option to do this FAST? I can write COBOL program.
Now I am trying to sort them using the below command, but it already run for a long time and just not going to finished.

Thank you.

The command I used: time sort -t ',' -o sorted.txt +0 -1 +4 -5 +5 -6 +6 -7 +22 -23 *.DAT_gprscdr_ftpd

share|improve this question

migrated from Feb 9 '11 at 12:51

This question came from our site for professional programmers interested in conceptual questions about software development.

Ask this question at – Amir Rezaei Feb 9 '11 at 7:50
Look at this SO question… you might be able to use the script in there. – KeesDijk Feb 9 '11 at 8:03
What's wrong with the shell script you provided? The unix sort is fast. – S.Lott Feb 9 '11 at 11:20
up vote 3 down vote accepted

Maybe this question really should be in another SE-site, but here is my take on this issue.

1) Isn't the basic sort you provided in your question fast enough? How fast it should be? My 2 year old desktop sorts 270MB of Apache access log files in 21 seconds.

2) If that is not fast enough, you can try to first sorting each file individually and then merge them with "sort -m"

3) If not fast enough and you have more than one CPU/core, parallelize (sp?) the process with GNU Parallel

4) If still not fast enough and you have more machines available, parallelize the sorting process on multilple machines with GNU Parallel

share|improve this answer

you can load the data in MySQL database (LOAD command) and do what you want

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .