Look for many different strings in many different log files?

Question

I have a list of files that have gone missing somewhere in our system at work. I also have a folder full of 41 log files adding up to 46 MB that hopefully have log entries pertaining to the missing files. How could I grep these log files for any value in my list?

The list is structured as one file per line without file extension. The logs do seem to have a structure, but I'm not entirely familiar with that structure yet. It does contain file names and paths as well as what was done to it.

I know I can cat * all the log files and pipe it to grep. I'll probably use -A and -B to get a little context out of the log files when a name is found. I'm using GnuWin32 on windows so I could couple this with Powershell but I think doing that would require that one filename greps all 46 MB and when I move to the next filename I start over. I have 1830 files in the list so if I have to start fresh with each one, I'll end up reading 46 MB so many times that I'll be dealing with GBs of repeating data. It seems inefficient to do it that way.

I suppose I could build a large regex of the 1830 files or'd together and run that once against the logs, but is that feasible? The regex would be almost 30KB (1830 files * average length of file name of about 16 chars = 29280 bytes not to mention another 1830 bytes of pipe symbols).

Edit: Here's what I'm doing now when I'm in the logs folder and the list is one folder back:

$logs = gc *
$notfound = gc ../notfound.txt
$logs | % { $i = 0; while ($i -lt $notfound.Count) { if ($_ -contains $notfound[$i]) { echo $_ }; $i++; } } | out-file C:\discovered.txt

It's entirely powershell. I'm up for using any tool to speed this up because right now there's 550991 lines in all the log files combined and there's 1830 filenames so this approach is making 1,008,313,530 comparisons. It's all in memory so at least I don't have disk I/O slowing me down. I might be able to break out of the while if the if becomes true but I am still going to make so many comparisons I'm not sure if optimizing that will actually do any good. It's already been running for half an hour. I'm ok with rewriting my approach from line 1 if I can get it done before I go home for the weekend.

Answer 1

It would be more efficient to pull the filenames out the logs via a regular expression and see if each of those is in your list. It could look something like this:

$notfound = gc ../notfound.txt
gc * |
        select-string -AllMatches '\\(?<filename>[^\\]+)\.txt' | 
        select -ExpandProperty Matches |
        % { $_.Groups['filename'].Value } |
        ? { $notfound -contains $_ } |
        out-file C:\discovered.txt

I am searching for files that look like "\something.txt". You will have to change that.

If it's still too slow and your notfound list is very large, then it might be more efficient to load it into a .Net HashSet, but I wouldn't do that unless needed.