2

For years Google News Alerts has sent daily emails when certain words pop up in websites, tweets, etc. and these emails (in HTML and .txt format) have one unique quality: there is no easy way to pull out the URL that contains where the Google News Alert keyword was found. As we literally have 1000's of these emails, we simply want to pull out the URL identified in a Google News Alert so that these URL's can be added to our google disavow list as well as our .htaccess file of domains we need to block.

Trying over and over again, we cannot get this .bat file to work. Simply it needs to have two delimiters set and where the batch file searches through a directory full of .eml/.html/.txt files and for each file that it looks into it pulls out the string that is in between url=3D and \u0026ct outputting said URL to a .txt file. From there we can just add the output (by hand 1x/mo or more if necessary) to our Google Disavow list and .htaccess block list.

Step 2, a complete automation where once the URL's are pulled out they are automatically added to the running disavow list and .htaccess temporary file that we put in the same directory can come next.

Where this gets VERY challenging is that since the MS DOS .bat file command to find strings (the findstr) command uses the usebackq command we think the findstr command has problems if there are any / and/or = characters in the search string itself.

Why this is a problem is that when parsing any of the Google News Alerts that are set (and we think Google does this on purpose) inside that email there is no way to parse out the start of the URL that needs to be grabbed out because the delimiter to show where the exact URL is starting is u0026url=3D thus the findstr command is failing.

As an example, here are two attempts where attempt #1 will pull out partial lines and attempt #2 will just return some strange output on every line of the DidIT.txt success file of:

   line:*=
   line:*=
   line:*=

which clearly shows a problem.

If anyone can help we would greatly appreciate this as we are trying to parse through 5000 Google News Alerts that we need to append to our block list and also our Disavow list.

Below are the two best examples of the .bat file we have tried and a simple snippet of the typical .html/.txt raw output of the emails that we get from Google News Alert emails.


Option #1

@echo off
setlocal enabledelayedexpansion

set "source_directory=D:\000\parser"
set "output_file=DidIT.txt"

REM Clear output file
type nul > "%output_file%"

REM Loop through all .eml files in the source directory
for %%F in ("%source_directory%\*.eml") do (
    echo Processing file: %%~nxF

REM Read each line of the .eml file
    for /f "tokens=2 delims==&" %%A in ('findstr /C:"url=3D" "%%F"') do (
        REM Extract characters between "&url=" and "&ct" delimiters
        for /f "tokens=1 delims=&" %%B in ("%%A") do (
            REM Output the extracted characters to the output file
            echo %%B>>"%output_file%"
        )
    )
)

echo Completed! The output has been saved to "%output_file%".

Option #2

@echo off
setlocal enabledelayedexpansion

set "directory=D:\000\parser"
set "outputFile=%directory%\DidIT.txt"

echo Processing .eml files in directory: %directory%

rem Delete the output file if it already exists
if exist "%outputFile%" del "%outputFile%"

for %%F in ("%directory%\*.eml") do (
    echo Processing file: %%~nxF
    set "delim1=u0026url"
    set "delim2=u0026cd"
    set "content="
    for /f "usebackq tokens=1,* delims=" %%A in ("%%F") do (
        set "line=%%A"
        if "!line:%delim1%=!" neq "!line!" (
            set "content=!line:*%delim1%=!"
            setlocal enabledelayedexpansion
            for /f "delims=%delim2%" %%C in ("!content!") do (
                echo %%C>> "%outputFile%"
            )
            endlocal
        )
    )
)

echo Output file didit.txt created successfully.

endlocal

Example Google News Alert .html/.txt/.eml email to run the above, make multiple copies and label them #1.eml, #2.eml, #3.eml

"widgets": [ {
"type": "LINK",
"title": "tmavomodr=C3=A9 =C5=w2q2wa blackblob, strih do A, obtiahnut=
=C3=A9, blabla",
"description": "yytree=C3=A1t =C4=8D. 814091234: tmiiiih=C3=A9 =C5=A1=
aty blackblob, strih do A, gheewse=C3=A9, wahwah, 234: 6 =E2=82=AC, L=
yayayay: Bansk=C3=A1 =C5=A0ooohooh.",
      "url": "https://www.google.com/url?rct=3Dj\u0026sa=3Dt\u0026url=3Dhtt=
ps://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=
1-2-333333-56543554.php\u0026ct=3Dga\u0026cd=111111111111111111111111111111=
222222222222222222222222222222222222222222222222222222\2322usg=erererererer=
3333333333333333333333333"
    } ]
  } ]
}
</script> <!--[if mso]>
 <table><tr><td width=342wq>
<![endif]-->
 <div style=3D"width:100%;max-width:650px"> <div style=3D"font-family:Arial=
"> <table style=3D"border-collapse:collapse;border-left:1px solid #121212;b=
order-right:1px solid #343434"> <tr> <td style=3D"background-color:#w2w2w2;=
padding-left:18px;border-bottom:1px solid #121212;border-top:1px solid #121=
332"></td> <td valign=3D"middle" style=3D"padding:13px 10px 8px 0px;backgro=
und-color:#w2w2w2;border-top:1px solid #q1q1q1;border-bottom:1px solid #111=

Output in DiDIt.txt should look like:

https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php

https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php

https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
1
  • 1
    Hi, any reason you have to do it in batch? There are so many better tools for such processing - e.g. Python. Google for sure doesn't target users of that old technology with URLs that are harder to parse... It's quite standard URL syntax.
    – Destroy666
    May 13, 2023 at 4:27

3 Answers 3

0

With this text in input.html

"widgets": [ { "type": "LINK", "title": "tmavomodr=C3=A9 =C5=w2q2wa blackblob, strih do A, obtiahnut= =C3=A9, blabla", "description": "yytree=C3=A1t =C4=8D. 814091234: tmiiiih=C3=A9 =C5=A1= aty blackblob, strih do A, gheewse=C3=A9, wahwah, 234: 6 =E2=82=AC, L= yayayay: Bansk=C3=A1 =C5=A0ooohooh.", "url": "https://www.google.com/url?rct=3Dj\u0026sa=3Dt\u0026url=3Dhtt= ps://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d= 1-2-333333-56543554.php\u0026ct=3Dga\u0026cd=111111111111111111111111111111= 222222222222222222222222222222222222222222222222222222\2322usg=erererererer= 3333333333333333333333333" } ] } ] } <!--[if mso]>

... a Python snippet to get you further:

C:\>python -c "import sys, re;s=' '.join(sys.stdin.readlines());z=re.findall('.0026url=3D(.+).u0026ct=3D',s);print(z[0])" <input.html

... which prints this from your (malformed?) example:

htt= ps://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d= 1-2-333333-56543554.php

This will "work" for @Destroy666's edited version in 2input.html:

python -c "import sys, re;s=''.join([line.replace('=\n','') for line in sys.stdin.readlines()]);z=re.findall('.0026url=3D(.+).u0026ct=3D',s);print(z[0])" <2input.html 

Output:

https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
4
  • thanks so much hannu but all emails from google are .eml files so it's raw text and not .html so we can only parse with .bat files or possibly powershell however one click .bat files from a file manager interface is the 100% no complications way to go ( no python to install, no powershell / powerscript complications to deal with ) and why we are pursing the 100% most basic way to solve this May 13, 2023 at 14:18
  • The data you provided is "HTML"-ish, and so is *.EML files - this due to the fact that Microsoft in it's wisdom (NOT!) started to use HTML as formatting for email - so the above should work just fine for *.EML files too... EML, HTML, VRML, and many more formats are just text, where the "ML" stands for "markup language" and indicates the use of "tags" e.g. <HTML> which is the leading tag in HTML.
    – Hannu
    May 13, 2023 at 15:20
  • trust me @hannu I'm very aware of the .EML format being the problem. no idea why every line in the 80th character slot ( in HTML sections of a google new alert email ) has to have '=' in there. If not the case, and it were just a simple line wrapped section or ( because you think they would do that ) all the spaces and line breaks, etc,. were in there so it could be read easily like any html file ) life would be a LOT easier. I'm sure you know what I'm talking about if you look inside any .eml file ya ? this really should be a super simple problem but, ah, .eml files are as they are May 16, 2023 at 13:09
  • line.replace('=\n','') in the 2input.html traversing code above takes care of the='s at-line-end problem.
    – Hannu
    May 16, 2023 at 15:52
0

Your example content for output string does not include the "=", in:

...d=1-2-333333-56543554.php

If this is not intentional (a typo) then you can use:

@echo off && setlocal enabledelayedexpansion

cd.>"DidIT.txt"
for %%G in ("D:\000\parser\*.eml")do echo/Processing file: "%%~nxG" && =;(
     for /f usebackq^tokens^=*delims^=^\ %%i in (`type "%%~fG" ^|find ".php"
        `)do for /f ^usebackq^delims^=^\ %%? in (`echo;%%~i`)do echo/htt%%~?
    );= >>".\DidIT.txt"

endlocal & echo Completed! The output has been saved to .\DidIT.txt
  • content saved in .\DidIT.txt:
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php

But to extract the string and remove the "=" in the output/file, you can try:

@echo off && setlocal enabledelayedexpansion

cd.>"DidIT.txt"
for %%G in ("D:\000\*.eml")do echo/Processing file: "%%~nxG" && =;(
    for /f usebackq^tokens^=*delims^=^\ %%i in (`type "%%~fG" ^|find ".php"
       `)do for /f ^usebackq^delims^=^\ %%? in (`echo;%%~i`)do set "_str=%%~?" && =;(
            for /f ^usebackq^tokens^=1*delims^=^= %%a in (`%ComSpec% /v /c echo;!_str!
               `)do >>".\DidIT.txt" echo/htt%%~a%%~b
            );=
        );=

endlocal & echo Completed! The output has been saved to .\DidIT.txt
  • content saved in .\DidIT.txt:
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php

An alternative in Powershell would be:

$str=$(gc -path "D:\000\*.eml" | sls ".php") | 
     ?{$_} ; $str=($str -replace ("=","") -split '\\u'|sls .php)
$str -replace ("ps:","https:") | out-file .\DidIT.txt
  • content saved in .\DidIT.txt:
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php

Or...

$str=$(gc -path "D:\000\*.eml" | sls ".php") | 
     ?{$_} ; $str=($str -split '\\u'|sls .php)
$str -replace ("ps:","https:") | out-file .\DidIT.txt
  • content saved in .\DidIT.txt:
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php

An alternative in Powershell would be:

$str=$(gc -path "D:\000\*.eml" | sls ".php") | 
     ?{$_} ; $str=($str -replace ("=","") -split '\\u'|sls .php)
$str -replace ("ps:","https:") | out-file .\DidIT.txt
  • content saved in .\DidIT.txt:
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=1-2-333333-56543554.php
9
  • Io-oI thanks for ninja level effort. fyi: only reason for batch is to run on any machine easily as in powershell, same problem '=' was getting in the way. FYI: '=' is appended to EVERY 80th character in any news alert sent by google. sure they could've used line wrap thus incredibly easy to parse, but nope, they send EVERY line with a '=' in the 80th character slot. any way to be able to have a variable for the 1st delimiter and the 2nd delimiter? in the above example we can't see where the "url=3D" and the "\u0026ct" is specified and thus i think this is where the error is coming from ? May 13, 2023 at 13:56
  • error ( from non powershell example above ) is " 'amp' is not recognized as an internal or external command, operable program or batch file. " and as I can't paste full email from 4 Google News Alert .eml files ( contain our google account information ) above example was a portion of whole email. the didit.txt output from above examples is: httqw-fredser-wsef-e-gfgvdfgdg344-dfdrrrt.php httfrrtrtgd-dfgrrf-rtrf-s-dftrf-rrfrtd.php htt571/gfgvrtrg-trtfrt-rtfrtgt-rtfrtrfrt-rtrfrt-fgrtrf.php htttgt-rtrtfrtrt.php where it's exactly pulling 4 correct url's from 4 files but mangling the URL May 13, 2023 at 13:56
  • and FYI: these emails ( when exported from g-mail: right click on 3 dots on message ---> download; from outlook, etc,. are exported either as .txt or in the case of Thunderbird, gmail, as .eml which is also a .txt based output ) thus they can't be parsed as .html which was the suggestion above. As .eml outputs there is the '=' at position 80 in each line. May 13, 2023 at 14:06
  • more specifically output goal is: x.org/zzzz ytrfrd.zz/freess/wedesd qq.com/qqq/tttt.mp3 # # domain:x.org domain:ytrfrd.zz domain:qq.com for our disavow list. Io-oI could you help? 1) a .bat file for above parsing; 2) a way to append URL's pulled out from above to be appended to an existing disavow list w/domain blocking added in & 3) RewriteCond %{HTTP_REFERER} aaa.bbb\.com [NC,OR] inserts to a .htaccess file for each URL pulled out? We need to easily lock this down & The more automatic the better w/goal of right click anywhere --> " send to disavow & block in .htaccess " is our goal May 13, 2023 at 14:24
  • Thanks for testing and for commenting, but sorry, it's very confusing. I suggest using a faithful file (extracting sensitive information obviously) and clearly defining your objective, without using syntax, but using an output example (expected layout).
    – Io-oI
    May 13, 2023 at 18:41
0

so from everyone's input, I've been able to put together the following code, yet it still does not work as it will not parse out the '=' that is inherent in every line in a .eml file inbetween the two delimiters that identify the URL of the actual web page that contains the key words in typical google news alert. did I miss something ?

@ioio @hannu Can you assist ?

@echo off
setlocal enabledelayedexpansion

set "prefix=\u0026url=3D"
set "suffix=\u0026ct=3Dga"
set "outputFile=output.txt"

> "%outputFile%" (
    for %%F in (*.eml) do (
        set "content="
        for /f "usebackq tokens=*" %%L in ("%%F") do (
            set "line=%%L"
            setlocal enabledelayedexpansion
            if "!line!" neq "" (
                set "content=!content!!line!"
                if "!line!"=="!suffix!" (
                    set "content=!content:*!prefix!=!"
                    set "content=!content:~0,-12!"
                    set "content=!content:=!"
                    echo !content!
                    set "content="
                )
            )
            endlocal
        )
    )
)

echo Output written to %outputFile%

endlocal

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .