Use wget. Simple, scriptable, command-line, and available on your favorite platform, whether it's Unix-ish, Win*, Cygwin, etc (see Wikipedia for links to various versions). From the manpage:
When invoked with this option, Wget will behave as a Web spider, which means that it will not download the pages, just check that they are there. For example, you can use Wget to check your bookmarks:
wget --spider --force-html -i bookmarks.html
This feature needs much more work for Wget to get close to the functionality of real web spiders.
You might want the
--output-document=file options too.