For years Google News Alerts has sent daily emails when certain words pop up in websites, tweets, etc. and these emails (in HTML and .txt
format) have one unique quality: there is no easy way to pull out the URL that contains where the Google News Alert keyword was found. As we literally have 1000's of these emails, we simply want to pull out the URL identified in a Google News Alert so that these URL's can be added to our google disavow list as well as our .htaccess
file of domains we need to block.
Trying over and over again, we cannot get this .bat
file to work. Simply it needs to have two delimiters set and where the batch file searches through a directory full of .eml
/.html
/.txt
files and for each file that it looks into it pulls out the string that is in between url=3D
and \u0026ct
outputting said URL to a .txt
file. From there we can just add the output (by hand 1x/mo or more if necessary) to our Google Disavow list and .htaccess
block list.
Step 2, a complete automation where once the URL's are pulled out they are automatically added to the running disavow list and .htaccess
temporary file that we put in the same directory can come next.
Where this gets VERY challenging is that since the MS DOS .bat
file command to find strings (the findstr
) command uses the usebackq
command we think the findstr
command has problems if there are any /
and/or =
characters in the search string itself.
Why this is a problem is that when parsing any of the Google News Alerts that are set (and we think Google does this on purpose) inside that email there is no way to parse out the start of the URL that needs to be grabbed out because the delimiter to show where the exact URL is starting is u0026url=3D
thus the findstr
command is failing.
As an example, here are two attempts where attempt #1 will pull out partial lines and attempt #2 will just return some strange output on every line of the DidIT.txt
success file of:
line:*=
line:*=
line:*=
which clearly shows a problem.
If anyone can help we would greatly appreciate this as we are trying to parse through 5000 Google News Alerts that we need to append to our block list and also our Disavow list.
Below are the two best examples of the .bat
file we have tried and a simple snippet of the typical .html
/.txt
raw output of the emails that we get from Google News Alert emails.
Option #1
@echo off
setlocal enabledelayedexpansion
set "source_directory=D:\000\parser"
set "output_file=DidIT.txt"
REM Clear output file
type nul > "%output_file%"
REM Loop through all .eml files in the source directory
for %%F in ("%source_directory%\*.eml") do (
echo Processing file: %%~nxF
REM Read each line of the .eml file
for /f "tokens=2 delims==&" %%A in ('findstr /C:"url=3D" "%%F"') do (
REM Extract characters between "&url=" and "&ct" delimiters
for /f "tokens=1 delims=&" %%B in ("%%A") do (
REM Output the extracted characters to the output file
echo %%B>>"%output_file%"
)
)
)
echo Completed! The output has been saved to "%output_file%".
Option #2
@echo off
setlocal enabledelayedexpansion
set "directory=D:\000\parser"
set "outputFile=%directory%\DidIT.txt"
echo Processing .eml files in directory: %directory%
rem Delete the output file if it already exists
if exist "%outputFile%" del "%outputFile%"
for %%F in ("%directory%\*.eml") do (
echo Processing file: %%~nxF
set "delim1=u0026url"
set "delim2=u0026cd"
set "content="
for /f "usebackq tokens=1,* delims=" %%A in ("%%F") do (
set "line=%%A"
if "!line:%delim1%=!" neq "!line!" (
set "content=!line:*%delim1%=!"
setlocal enabledelayedexpansion
for /f "delims=%delim2%" %%C in ("!content!") do (
echo %%C>> "%outputFile%"
)
endlocal
)
)
)
echo Output file didit.txt created successfully.
endlocal
Example Google News Alert .html
/.txt
/.eml
email to run the above, make multiple copies and label them #1.eml
, #2.eml
, #3.eml
"widgets": [ {
"type": "LINK",
"title": "tmavomodr=C3=A9 =C5=w2q2wa blackblob, strih do A, obtiahnut=
=C3=A9, blabla",
"description": "yytree=C3=A1t =C4=8D. 814091234: tmiiiih=C3=A9 =C5=A1=
aty blackblob, strih do A, gheewse=C3=A9, wahwah, 234: 6 =E2=82=AC, L=
yayayay: Bansk=C3=A1 =C5=A0ooohooh.",
"url": "https://www.google.com/url?rct=3Dj\u0026sa=3Dt\u0026url=3Dhtt=
ps://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d=
1-2-333333-56543554.php\u0026ct=3Dga\u0026cd=111111111111111111111111111111=
222222222222222222222222222222222222222222222222222222\2322usg=erererererer=
3333333333333333333333333"
} ]
} ]
}
</script> <!--[if mso]>
<table><tr><td width=342wq>
<![endif]-->
<div style=3D"width:100%;max-width:650px"> <div style=3D"font-family:Arial=
"> <table style=3D"border-collapse:collapse;border-left:1px solid #121212;b=
order-right:1px solid #343434"> <tr> <td style=3D"background-color:#w2w2w2;=
padding-left:18px;border-bottom:1px solid #121212;border-top:1px solid #121=
332"></td> <td valign=3D"middle" style=3D"padding:13px 10px 8px 0px;backgro=
und-color:#w2w2w2;border-top:1px solid #q1q1q1;border-bottom:1px solid #111=
Output in DiDIt.txt
should look like:
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php
https://ohohohoh.wawawa.zz/ghhhhyh656y/deas/yayaya-iiiuiiu-dd1111111dee-yaya-d1-2-333333-56543554.php