2

If some domain name becomes stale, it usually still replies to HTTP requests with some bunches of useless links, sometimes even without an explicit "This domain is expired/for sale" message instead of just failing to resolve at all.

How to automatically determine the "dead" (expired/not prolonged) domain without a browser? Can it be done with whois tool?

Expecting something like this:

while true; do
    if ! checkdomain something-on-verge-of-abandoning-business.com; then
        echo "Good night, sweet prince"
        break
    fi
    sleep 1d
done

For example, let's look at domain allmydata.com. As far as I see about Allmydata, there should be some Tahoe-LAFS-based backup service there. But actually it is parked domain.

$ whois allmydata.com
...
   Domain Name: ALLMYDATA.COM
   Registrar: DOMAIN MONKEYS, LLC
   Whois Server: whois.domainmonkeys.com
   Referral URL: http://www.domainmonkeys.com
   Name Server: NS1.DSREDIRECTION.COM
   Name Server: NS2.DSREDIRECTION.COM
   Status: clientTransferProhibited
   Updated Date: 04-aug-2013
   Creation Date: 03-aug-2004
   Expiration Date: 03-aug-2014
...

$ dig +short -t A allmydata.com @8.8.8.8
208.73.211.247

This shallow check shows like it is were a good domain. But how to reliably (i.e. not heuristically parsing the page and measuring "spammy-ness" or "parked-ness" from content) detect such thing?

Improve this question
2
  • linux I assume?
    – Journeyman Geek ♦
    Dec 5, 2013 at 11:00
  • For example, GNU/Linux. But I expect the approach to be portable.
    – Vi.
    Dec 5, 2013 at 16:45

1 Answer 1

Reset to default
1

You could just use python to read in the html content and then search for "domain available"/"parked"/"renewal" etc and other keywords. You can feed it a CSV file of domains and then output the results as CSV and there you have a list of domains.

The other idea would be to parse whois records using something like this and parse the results for the renewal date. That's how I'd do it.

Improve this answer
4
  • This looks like a workaroundish "heuristic" method... What is the essential technical difference between normal and "dead" domain?
    – Vi.
    Dec 5, 2013 at 12:13
  • Well a dead domain will have no valid nameserver or A records - you could just dig for those and use that as a test/fail thing, otherwise I'm out of ideas. Personally I would just query whois records and parse the results for the renewal date - any date that <= today would be a dead domain
    – Mud
    Dec 5, 2013 at 13:46
  • Appended a domain example that satisfies "expire date > today", but yet still a parked domain.
    – Vi.
    Dec 5, 2013 at 16:46
  • Well this wouldn't show parked domains because a parked domain is just a domain with a holding page on it. To check for a holding page you would have to read in the source code of the site and search for obvious strings like "holding page" "parking" "domain" and maybe grab a list of the top 200 hosts and iterate through that. My solution above is only for domains that have expired or are close to expiring
    – Mud
    Dec 6, 2013 at 15:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .