Performing PowerShell analog of *nix grep <multiple files by mask>

Question

Is there a way to optimize PowerShell code below (it greps particular lines by string contained from a bunch of text files into a single one):

$ErrorActionPreference = "Continue"
Start-Transcript -path D:\0xAC1CC07A.log -append
$OutFile = "D:\0xAC1CC07A.txt"
echo "filtering 0xAC1CC07A"
ForEach ($filenm in ((get-childitem -Path D:\FILES\* -include ubuntlive1mb_?????_201509*.txt -recurse -force))) 
{
 $filenm.fullName;
 (Get-Content $filenm) | select-string "0xAC1CC07A" | Add-Content $OutFile
}
Stop-Transcript

It does well on small workloads but on 160K text files (over 200GB in total) it works more than 4 days on my Win2008R2 VM. Surprisingly Ubuntu 14.04 on the similar virtual hardware did the job within 4 hours:

grep --no-filename "0xac1cc07a" ./FILES/ubuntlive1mb_?????_201509*.txt >>./0xAC1CC07A.txt

Or more precisely:

find ./FILES -name "ubuntlive1mb_?????_201509*.txt" -type f -print0 | xargs -0 grep --no-filename "0xac1cc07a" $1 >>./0xAC1CC07A.txt

I am neither good at PowerShell nor at *nix, all above scripts were created by googling and copy-pasting.

Windows box has been file-system optimized by disabling dos file names and directory update on list. Ubuntu was just installed out of the box.

Answer 1

This very simple Powershell script should do what you're looking for:

$OutFile = "D:\0xAC1CC07A.txt"
Get-ChildItem -Path D:\FILES\ubuntlive1mb_?????_201509*.txt -Recurse | Foreach-Object { Select-String -Path $_ -Pattern "0xAC1CC07A" } | Foreach-Object { Add-Content -Path $OutFile -Value $_.Line }

This will just add the matched lines into the $OutFile text file. You could also use this to get the file names or the line numbers of the matched lines as well, by using the Filename, Path, and LineNumber properties, instead of just the Line property.

If you want to test a script which will run against many files, but don't want to wait for it to finish checking all of them, then you can use the Select-Object cmdlet to limit the number of files it will check.

Example:

Get-ChildItem -Path D:\FILES\ubuntlive1mb_?????_201509*.txt | Select-Object -First 100 | Foreach-Object { Select-String -Path $_ -Pattern "0xAC1CC07A" } | Foreach-Object { Add-Content -Path $OutFile -Value $_.Line }

This will run the above script only against the first 100 text files that are returned from Get-ChildItem.

Answer 2

You'll have slightly different output (but that can be taken care of should there be a need) but from what I've seen it's quite a bit faster just going for the Select-String directly on the file instead of getting the file contents first.

Select-String "0xAC1CC07A" -Path $filenm.FullName | Add-Content $OutFile

Just remember to check the output first before appending it to file so you get it in the way you desire.

As for speed; Get-ChildItem is notoriously slow in PowerShell (since PowerShell likes to fetch objects rather than just text representation of objects) and there are various workarounds for this.

The Get-ChildItem-line in your code can be optimized however. From what I've seen using Filter is roughly 3,5 times faster than using includes/excludes on a regular consumergrade 7.2k HDD.

Get-ChildItem -Path "D:\FILES" -Filter "ubuntlive1mb_?????_2015090101*.txt" -Recurse -Force

If memory serves me right, earlier versions of PowerShell had some problems with filter, such as if you wanted to all files with extension .htm it would also pick up the files with the extension .html (as if you had filtered *.htm* and not *.htm), so you might wanna keep an eye out for that.