0

I made a script to help me analyze the access log files without SSH to the server (I have the file only). It counts and sorts the number of IP addresses pounding sites I manage, but I've found it takes much too long on large files (it is VERY rudimentary). I didn't want to have to use a compiled application and don't have SSH to the server, so I turned to Powershell.

$sw = [Diagnostics.Stopwatch]::StartNew()
$input_path = ‘c:\temp\access_log2’
$ip_file = ‘c:\temp\IPs.txt’
$output_file = ‘c:\temp\SORTED.txt’
$regex = '\b(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\b'
select-string -Path $input_path -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $ip_file
get-content $ip_file | group-object -noelement | Sort-Object count -descending > $output_file
get-Content $output_file -First 25
$sw.Stop()
$sw.Elapsed

I also tried

$regex = ‘\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b’

on 5MB file(14.4K lines), it took 18 minutes on an 37MB file (158.5K lines), it took over 3 hours

The stopwatch is just for my testing. The script is just to pull the IP, count them, and sort by most occurring. Maybe the file writing is the biggest slowdown but I'm not too familiar with variables stored in RAM. I think there's a better way to extract IP addresses (maybe just go with first 15 characters per line?). Here is an example of the lines, Combined Log Format

21.198.52.3 - - [06/Aug/2017:11:31:54 -0400] "GET / HTTP/1.0" 301 452 "-" "-"
154.212.178.24 - - [06/Aug/2017:11:10:44 -0400] "GET /images/12345.jpg HTTP/1.1" 200 212443 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"

Any advice?

  • if the log lines always start with a valid IP why use a complex RegEx to check the validity? – LotPings Aug 10 '17 at 18:57
  • They do, can I use other .net functions like IPAddress.TryParse as I do for validating IPs for c#? – loserspearl Aug 10 '17 at 20:29
0

There is a simpler (hopefully faster approach) if the ip is leading every line. A simple split at the space should seperate the ip.

While processing the lines you can increment a hashtable addressed by the ip

$sw = [Diagnostics.Stopwatch]::StartNew()
$input_path  = 'c:\Temp\access_log2'

$Hash = @{} 
ForEach ($Line in (Get-Content $Input_path)) {
    ++$Hash[$Line.split(' ')[0]]
} 
$Hash.GetEnumerator() | sort -Property Value -desc |Select -First 25
$sw.Stop()
$sw.Elapsed

Sample output with only your lines replicated:

Name                           Value
----                           -----
154.212.178.24                 5
21.198.52.3                    4

Ticks             : 121118
Days              : 0
Hours             : 0
Milliseconds      : 12
Minutes           : 0
Seconds           : 0
TotalDays         : 1,4018287037037E-07
TotalHours        : 3,36438888888889E-06
TotalMilliseconds : 12,1118
TotalMinutes      : 0,000201863333333333
TotalSeconds      : 0,0121118
  • Count Name ----- ---- ...52 127.171.123.44 Name Value ---- ----- 127.171.123.44 145214 This doesn't cutoff the count value when it gets over 6 digits long, nor does it grab an agent IPs that could be down the row of the log record, noice – loserspearl Aug 10 '17 at 22:30
0

Why are you writing your data to an intermediate file ($ip_file)? Why not pass it directly from to the Group-Object? Perhaps something like this?

$sw = [Diagnostics.Stopwatch]::StartNew()

$input_path  = ‘c:\temp\access_log2’
$ip_file     = ‘c:\temp\IPs.txt’
$output_file = ‘c:\temp\SORTED.txt’
$regex       = '\b(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\b'

Select-String -Path $input_path -Pattern $regex -AllMatches |
  ForEach-Object { $_.Matches } |
  ForEach-Object { $_.Value } |
  Group-Object -noelement |
  Sort-Object count -descending > $output_file
Get-Content $output_file -First 25

$sw.Stop()
$sw.Elapsed

If that doesn't make things faster then you should probably try and figure out which part of your script is the slow part.

Maybe start by just doing something like this. Then add functionality back a bit at a time to see where the slowness is.

$sw = [Diagnostics.Stopwatch]::StartNew()
$input_path  = ‘c:\temp\access_log2’
$regex       = '\b(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\b'
Select-String -Path $input_path -Pattern $regex -AllMatches | Out-Null
$sw.Stop()
$sw.Elapsed

If you are willing to assume that the log file is always properly formated you could probably speed things up and just grab the first column. (This processed a 233MB, 1,000,749 line apache log file on my computer in about 15 seconds)

Get-Content c:\temp\access_log2 | ForEach-Object { $_.split(' ')[0] } |
Group-Object -NoElement |
Sort-Object Count
  • I'm not too familiar with PS so I started by just getting it workinmg. The new piping works much better. Also, changed to a speedier HDD and it gets it down to a few minutes. TY – loserspearl Aug 10 '17 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.