1

There is a plain (html) web page, which occasionally gets updated with text information. Is there any way to get notified (by any means) if this particular web page content is changed whereas website doesn't provide any subscription mechanism?

1
  • periodically programatically scrape the site, record the MD5/SHA sum, compare to previous invocation and send an alert/email when it changes. In Linux you can use curl or wget or a number of other tools, and wrap it in a trivial bash script. – davidgo Apr 21 '20 at 8:24
1

Linux (and WSL)

You should be able to use the following even in the Windows Subsystem for Linux (WSL)

Under Linux you can use wget with the -N option (timestamp) from the directory in which you have previously downloaded the page.

wget -N  https://example.com/your_page.html 

It will download only a new version of the file, asking to the server the timestamp. wget will give a different output in case the download succeed or not.

  • File downloaded

    HTTP request sent, awaiting response... 200 OK

  • Not downloaded because it has the same timestamp

    HTTP request sent, awaiting response... 304 Not Modified

So at the end you can build your command-line or a script similar to

wget -N https://example.com/your_page.html 2>&1 \
      | grep "304 Not Modified" >/dev/null \
      && echo "Not downloaded, old one" \
      || echo "There is a new file"

Notes:

  • -N activate the timestamp usage
  • 2>&1 redirect the standard error to the standard output, needed for the following grep
  • | grep "304 Not Modified" pipe the output of the previous command (wget) as input of grep that selects the line with "304 Not Modified". It exit with status 0 (true) if it founds a match.
  • && logic and (aka then) execute the following command if the previous exit status is 0. You can omit this part.
  • || logic or (aka else) execute the following command if the exit status is not 0.
  • each \ is used in the shell to go to a new line and it must be the last character of the line
  • change echo this or that with your command...

Final remarks

With wget -O - | md5sum you can create a checksum to reuse the next time without the need to keep a local copy...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.