0

I have dozens of folders which contains both plain text log files and gzipped older logs. My goal is just a one liner to run (one folder at a time) to pull up all grep results based on timestamp order, regardless if the log file containing a match is txt or gz, and if possible to optimize for performance.

This works fine for the plain files:

ls -rt log.*.txt | xargs grep <treasure> -

I use this rather than just grep, so the results are sorted in chronological order of file creation, possibly spanning multiple days, rather than sorted based on the filename. Filenames (log.#.txt) grow to a certain integer limit, then wrap to log.0.txt, but this can cross over the 24hr mark or not.

Once the txt files wrap, the older files are gzipped: log.#.archive.gz. Only integer-limit gz files are kept.

I thought to use an if/fi statement to either grep or zgrep depending on the current file's extension. However, my first step to try it on just the gz files didn't work:

ls -rt log.*.gz | xargs zgrep <treasure> -

I get a bunch of errors "file 'treasure' not found" (one for each .gz file)

I've also tried

ls -rt "log.*.gz" | xargs -0 zgrep <treasure> -

to the same result. I known this must be due to my rudimentary understanding of the xargs command. Maybe I can even accomplish this with proper grep/zgrep options, find, or something else entirely.

  • Does <treasure> have any spaces or special characters in it ? – davidgo Jan 1 '15 at 8:46
  • In my version of zgrep you get identical output to that of grep if you give it a regular file instead of a .gz, so maybe you can just use that by default. From man zgrep: "Otherwise the given files are uncompressed if necessary and fed to grep." (emphasis mine) – WAF May 17 '16 at 15:51
0

Here are a few things wrong:

  • Try not to iterate over or pipe output produced by ls into another tool. It will break if files contain whitespace or newlines, depending on how the command is constructed. In your case, however, there's no easy way to accomplish what you want other than using ls. So if you know that your file names will not contain a newline, then you're good.

  • <treasure> - will be interpreted by the shell as redirection. The first bracket < will be read by the shell, meaning "read STDIN from a file called treasure". The second bracked would be read as > -, that is, "write STDOUT to a file called -". So, you should properly quote the pattern you are trying to read: grep "<treasure>".

  • The -0 option to xargs reads input as ASCII NUL-delimited lines, which ls will not produce. It is only useful in combination with tools that can create NUL-delimited output, such as find with the -print0 option.

  • I don't understand the purpose of - in your commands.

So, try something like this:

shopt -s extglob
ls -rt1 +(log.*.txt|log.*.gz) | xargs zgrep "<treasure>"

or:

ls -rt1 +(log.*.txt|log.*.gz) | xargs -L1 zgrep "<treasure>"

Explanation:

  • extglob allows the matching of both file extensions
  • -1 makes ls output one file per line
  • +(…|…) means "one or more" of the pattern
  • If you use -L1, xargs will pass only one file to zgrep at a time. This may not be what you want though.
  • Thanks, filenames is only alphanumerics and periods, search pattern only alphanumerics (should've left out the brackets) so it seems for loop also works. But this is more compact and explicit that it searches for both extensions. – user2632063 Jan 1 '15 at 23:42
-1

What about:

for each in `ls -rt log.*.gz`; do zgrep "<TREASURE>" $each; done
  • This breaks if file names contain whitespace or globbing characters of any kind. – slhck Jan 1 '15 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.