0

I have several folders that contain numerous text files, ranging from tens to 100s. These text files are simple databases containing millions of lines, with each line containing a single record. However, the records in them are unsorted and contain many duplicates. I'd like to to sort and de-duplicate them all individually (i.e. independently of each other), but to my understanding, sort can only produce a concatenated output of all input files - that is, even if given multiple files, it will only produce one output file containing the combined results of all those files.

How can I sort all files in the current folder to produce an individually sorted output file for each one? I'd also like for the output files to be outputted to a subfolder within the current directory. A for loop is the obvious solution to me, but I'm asking here in case there's some simpler way to do this with sort that I've not come across or missed. My bash knowledge is also very lacking, so if a for loop is the simplest solution, I'd appreciate someone providing the best way to go about that rather than me spending many days hacking something together that would still fall short of what I want to do.

  • sort -u provides a single record list. – matzeri Sep 13 '18 at 9:26
  • @matzeri Are you trolling? – Hashim Sep 13 '18 at 16:29
1

Yes, you can do this with for. Even if there is "some simpler way to do this with sort" (but I don't think so), this is also quite simple:

# cd to the directory you want to process

mkdir sorted    
for file in *; 
do
   printf 'Processing %s\n' "$file"
   [ -f "$file" ] && sort -u "$file" > "./sorted/$file"
done

Notes:

  • for file in * doesn't process files in subdirectories.
  • printf is only to report progress. In fact it should be placed after [ ... ] (see below) but I don't want to overcomplicate the code. You can just remove the printf line, if you want the whole thing to be silent.
  • [ -f "$file" ] checks if $file is a regular file. With the most general pattern (i.e. *) we need this condition at least to avoid running sort with the sorted directory as an argument (this would throw an error, harmless but non-elegant). Most likely this test won't be needed if you use a more specific glob like *.txt or *.db instead of * (e.g. to skip a stray desktop.ini file that shouldn't be processed). In this case you can omit [ ... ] && and start the line with sort (leaving the line intact shouldn't hurt though).
  • sort supports various options and you may want to use some of them, depending on how you need to sort.

  • sort -u de-duplicates entries right after sorting them, and when already using sort is a less redundant alternative to using the uniq command.

If you needed to pick files according to conditions more complex than a simple glob, find might be better to start with. For your current task for should be fine.

  • This looks perfect, thank you so much. There is a part of the code that I'm a little confused on - where does the [ -f "$file" ] come from? That is, what command does the -f parameter belong to, and what exactly do the square brackets mean? I haven't seen that sort of syntax used in a bash file before, at least not in this context. – Hashim Sep 13 '18 at 19:09
  • Okay, I found my answer here that it's synonymous with the test command: stackoverflow.com/questions/28036052/what-does-f-mea, but for clarification, what exactly does this mean? That [ ... ] is shorthand for the test command? – Hashim Sep 13 '18 at 19:22
  • @Hashim See help [ in Bash. – Kamil Maciorowski Sep 13 '18 at 19:41
  • Unfortunately that doesn't give any information other than saying it's a synonym, so I suppose that's the only answer. There is one more thing - I've become accustomed to doing LC_ALL=C sort to make what usually be a ridiculously slow sort command a lot faster by limiting it to the default (US English) locale. Do you know of any way to incorporate that into your code above? Simply adding it before sort as normal results in a syntax error near unexpected token &&. It's not a major problem if it can't be done, but it would be nice to find out whether it can be. – Hashim Sep 13 '18 at 21:20
  • 1
    @Hashim Yes. When the shell parses printf, it has no reason to look forward to the next line. When the shell encounters your &&, the line with printf has already been parsed and executed. – Kamil Maciorowski Sep 13 '18 at 21:51

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.