Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to write a shell script to grab the dynamic URL which ComboFix is located at at BleepingComputer.com/download/combofix

However, for some reason I can't seem to get my regex to match the download link of the "click here" if the download doesn't work. I used a regex tester and it said I matched the link, but I can't seem to get it to work when I execute it, it turns up an empty result. Here's my entire script:

 #!/bin/bash
 # Download latest ComboFix from BleepingComputer
 wget -O Listing.html "http://www.bleepingcomputer.com/download/combofix/" -nv
 downloadpage=$(sed -ne 's@^.*<a href="\(http://www[.]bleepingcomputer[.]com/download/combofix/dl/[0-9]\+/\)" class="goodurl">.*$@\1@p' Listing.html)
 echo "DL Page: $downloadpage"
 secondpage="$downloadpage"
 wget -O Download.html $secondpage -nv
 file=$(sed -ne 's@^.*<a href="\(http://download[.]bleepingcomputer[.]com/dl/[0-9A-Fa-f]\+/[0-9A-Fa-f]\+/windows/security/anti[-]virus/c/combofix/ComboFix[.]exe\)">.*$@\1@p' Download.html)
 echo "File: $file"
 wget -O "ComboFix.exe" "$file" -nv
 rm Listing.html
 rm Download.html
 mkdir Tools
 mv "ComboFix.exe" "Tools/ComboFix.exe" -f

The first two downloads work successfully, and I end up with: http://www.bleepingcomputer.com/download/combofix/dl/12/

But it fails to match the final sed that will give me the download link.

The code it's supposed to match is:

<a href="http://download.bleepingcomputer.com/dl/6c497ccbaff8226ec84c97dcdfc3ce9a/5058d931/windows/security/anti-virus/c/combofix/ComboFix.exe">click here</a>

SOLUTION:

For anyone interested, the final code that worked was:

#!/bin/bash
# Download latest ComboFix from BleepingComputer
wget -O Download.html "http://www.bleepingcomputer.com/download/combofix/12" -nv
file=$(sed -ne 's@^.*<a href=\x27\(http://download[.]bleepingcomputer[.]com/dl/[0-9A-Fa-f]\+/[0-9A-Fa-f]\+/windows/security/anti[-]virus/c/combofix/ComboFix[.]exe\)\x27>.*$@\1@p' Download.html)
echo "File URL: $file"
wget -OSN "ComboFix.exe" "$file" -nv
share|improve this question
up vote 1 down vote accepted

That's funny: I just went to http://www.bleepingcomputer.com/download/combofix/dl/12/ and found

<a href='http://download.bleepingcomputer.com/dl/587c6038..../..../ComboFix.exe'>click here</a>.

i.e., the URL is delimited by single quotes (') rather than double quotes (") (which is what your sed command is looking for).

share|improve this answer
    
How very strange, I'm viewing it in Chrome and I'm seeing double quotes – Eckster Sep 19 '12 at 2:47
    
You're right though, that's the problem, I viewed the Download.html file and it has single quotes – Eckster Sep 19 '12 at 2:53

There is no need to download the first page as the second page is static. It will always be:

http://www.bleepingcomputer.com/download/combofix/dl/12/

It is also strongly suggested that you use the -N flag on wget to check timestamps. This will only download ComboFix if its newer then the one on your computer.

Also make sure your script only runs once or twice a day. There is no need to run it more than that.

By limiting the requests and using the -N, you should avoid being automatically banned by our scripts.

Thanks

share|improve this answer
    
I wasn't certain if the first one was static or not, since otherwise the number seems unnecessary, thanks for letting me know. Currently the script only runs once a day at night, I'll be sure to add the -N to it, thanks for the tip and inside information :) – Eckster Sep 20 '12 at 7:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .