I have crons that does a name-check with the dns-server hourly. Nearly 300 machines run these crons locally. The cron is a shell script that tries to resolve ip address and hostname of the machine it is running on. It compares results from different command like for example :
dig + short $(hostname) host $(hostname -s) | head -1 | fex -1
If any of the comparison fails, it sends an email complaining that DNS check failed for the machine.
Recently and randomly, I am getting couple of machines that fails to resolve one of these checks. This happens let's say 10 machines per week. Many times it ends up with this kind or error:
"connection timed out; no servers could be reached"
Is there a way to catch these random events without clogging the server ?
I am dealing with mixed RHEL-5 and RHEL-6 machines.