0

I am trying to write a bash script that receives a path to a directory as its argument and outputs all lines of all log files (from the given directory and all its sub-directories) sorted by a primary key and a secondary key. sort by primary key needs to be in LINEAR TIME (n being the total number of lines); there's no run-time limitation for secondary sort.

EDIT: the output lines must have the exact same structure as in the original files, so any change to the lines/files needs to be reversed back to the original form. I can't change the log files format since I don't have access to them in advance. The amount of input files can be a few tens of thousands.

-the primary key is a timestamp of the following format: 2018-07-23T14:66:55.456789123

-the secondary key should be used for lines with identical timestamp; the secondary key is the path of each such line, from the input directory to the file that it came from. sorting by this key is done lexicographically.

- all log files have a ".log" extension

- all individual log files are already sorted in ascending order

- each log line in a log file has the following format: it starts with a timestamp, followed by a single space and some text, and terminates with \0. the text in each line can contain \n

EXAMPLE:

Below are 3 example files:

$ cat input_dir/A/f3.log

2019-08-22T13:33:44.123456789 get

up 2

2020-01-01T11:22:33.123456789 love 2

$ cat input_dir/B/f2.log

2019-09-44T13:44:21.987654321 Nice line

$ cat input_dir/C/f1.log

2019-08-22T13:33:44.123456789 get

up

2020-01-01T11:22:33.123456789 love

Here is my most successful attempt to solve the task:

find $1 -name "*.log" -printf '%p\n' | sort | xargs sort -r -m -t "-" -k1,1 -k2,2 -k3,3

Here is the output:

2019-09-44T13:44:21.987654321 Nice line

2019-08-22T13:33:44.123456789 get

up 2

2019-08-22T13:33:44.123456789 get

up

2020-01-01T11:22:33.123456789 love 2

2020-01-01T11:22:33.123456789 love

you can see that apart from the first line, it sorts correctly, and it looks like the sort command only compares 2019 to 2020 as keys...

I have also tried an incredible amount of possible combinations of flags for sort, including specifying different columns(-k), separators(-t), that seemed to have no effect on the sorting, probably due to my use of -m flag, but I can't avoid it since it makes the run-time linear and also makes it a stable sort path-wise. Also, I tried using -z flag to make the multi-lines stay together, but when I used it, it only sorted by timestamp and ignored the sorted paths I am trying to pass to the sort command.

I don't know if it's even possible to solve this task with simply using sort command, but I don't know how else to do it. does anyone have an idea how to solve this (in bash)? THANK YOU

0

I surprised to not see a -z parameter anywhere (required if your separator is \0), and for me your separator is a space and you only have to sort on the first item, which is the whole timestamp.

You have to inject the file name in the file itself (after the time stamp) so that sort can use it to disambiguate identical time stamps. With a known number of files this can be done like this:

sort -m -k2 -k1 <(sed 's/^/file1 /' file1) <(sed 's/^/file2 /' file2)

Where:

  • each sed sub-shell prefixes the file with a path
  • sort is performed on time stamp first (k2) and then on the filename (k1)

For a variable number of files you would have to generate the command dynamically and run it with eval or bash -c.

But:

  • if your timestamps are down to the nano second you'll almost never encounter two identical timestamps so this could be superfluous.
  • if this is your app you can also tweak the log format to inject a name and avoid the sub-shells.
  • in modern apps everybody is using some form of the ELK stack.
  • Thanks for the great idea, the problem is that I must have the output lines look exactly the same as they did in the original files. As for using ELK or tweaking, unfortunately I don't have the files nor do I know how many files my script will have to handle(could be a massive amount) as this is a school assignment(I have only learned bash last week and been trying to solve it everyday since then). Also, there will probably be identical timestamps just to test if my script does what it supposed to... – Tera Byte Nov 10 at 18:30
  • about the -z you're definitely right, I used it at first, but the result was sorted only by timestamp which made me think -z was making sort "ignore" my previous sorting of the paths... now I realize that maybe sort -m just ignores the order in which the files is given to it when it does the sort :/ – Tera Byte Nov 10 at 18:44
  • You pipe the output into a final sed stage to remove the paths you added. And yes, experimentally sort -m doesn't care about the order of the file parameters. If you get a clever answer to this later please post in a comment (or even in an answer...) – xenoid Nov 10 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.