Check a list of domains with the WHOIS command

Question

I want to set up a batch file or cronjob of some sort, using the linux command line, to check the availability of domain names. I will provide the list of domains to check using another file. It must then only do 30 checks per hour, and only if the domain is available, it must add that domain name to a different file. So...

1. Do I have to create a batch file?
2. What command must go in the batch file, that will:
   1. move through an existing list of domains
   2. check if those domains are available,
   3. remove that domain from the original list, and
   4. add the available domain to the "available" list.
3. Repeat this command 20 times every hour

I know exactly how to do this with PHP and cron, but I want to know know if there is a quick way to do this using the "whois" command? It's point (2) that I am stuck on.

Answer 1

That's not too hard.

Check the list of domains domains.txt, and add the domain to available.txt once it's found (by looking at the return value of grep, which is stored in $?).

Then, remove the found domains from domains.txt with sed in-place editing.

#!/bin/bash

AVAILABLE=~/available.txt
DOMAINS=~/domains.txt

lockfile whois-script.lock

while read -r domain; do
  whois $domain | grep -qci "No match"
  if [ $? -ne 0 ]; then
    # found
    echo $domain >> $AVAILABLE
  fi
done < $DOMAINS

while read -r domain; do
  sed -i "/$domain/d" $DOMAINS
done < $AVAILABLE

rm -f whois-script.lock

Note: On BSD sed, you want to use the following command:

sed -i "" "/$domain/d" $DOMAINS

You can save this script and call it from your crontab. Enter

crontab -e

and then add a line like this:

*/2 *   *   *   *   /path/to/script.sh > /dev/null

This will run the script every two minutes (*/2). Make sure to adjust the paths to the domain before.

Answer 2

Why go through all the trouble? You can now purchase all the whois data you'd like. I'm not saying it'll be cheap (for example https://alldomainsdb.com/) but it'll save you a lot of hard work. But that is if you need huge amount of whois data. If 30 checks per hour suffice, go ahead and make the batch file.