0

Certain network based programs recently (problem did not exist a couple of weeks ago) started hanging for me. Some that I've experienced the problem with are mysql, ssh, rsync, and git. My first thought was a network connectivity problem, but pinging the hosts in question (webserver and mysql server hosted by godaddy.com) showed they were live. (I used the domain name with ping in both cases.) At some point I tried using the IP address of the hosts directly and both programs worked as expected.

I haven't tried connecting these programs to other hosts.

I did an strace on mysql to see where it was hanging and here's the tail of that output:

stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=192, ...}) = 0
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
poll([{fd=3, events=POLLOUT}], 1, 0)    = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "\201\226\1\0\0\1\0\0\0\0\0\0\16mendelssohnalb\2db\0073"..., 62, MSG_NOSIGNAL, NULL, 0) = 62
poll([{fd=3, events=POLLIN}], 1, 5000)  = 1 ([{fd=3, revents=POLLIN}])
ioctl(3, FIONREAD, [78])                = 0
recvfrom(3, "\201\226\201\200\0\1\0\1\0\0\0\0\16mendelssohnalb\2db\0073"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.1")}, [16]) = 78
close(3)                                = 0
open("/etc/gai.conf", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3343, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=3343, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8a686a000
read(3, "# Configuration for getaddrinfo("..., 4096) = 3343
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7fe8a686a000, 4096)            = 0
futex(0x7fe8a5ad14a0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
socket(PF_NETLINK, SOCK_RAW, 0)         = 3
bind(3, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(3, {sa_family=AF_NETLINK, pid=2640, groups=00000000}, [12]) = 0
sendto(3, "\24\0\0\0\26\0\1\3\255\241WS\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0\255\241WSP\n\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 108
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\255\241WSP\n\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128
recvmsg(3, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\255\241WSP\n\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
close(3)                                = 0
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET6, sin6_port=htons(3306), inet_pton(AF_INET6, "::ffff:67.215.65.145", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
getsockname(3, {sa_family=AF_INET6, sin6_port=htons(38834), inet_pton(AF_INET6, "::ffff:192.168.0.104", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
connect(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(3306), sin_addr=inet_addr("97.74.31.26")}, 16) = 0
getsockname(3, {sa_family=AF_INET6, sin6_port=htons(56707), inet_pton(AF_INET6, "::ffff:192.168.0.104", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
close(3)                                = 0
socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET6, sin6_port=htons(3306), inet_pton(AF_INET6, "::ffff:67.215.65.145", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
+++ killed by SIGINT +++

From this it looks like a getaddrinfo(3) problem. It looks like its trying to connect using IPv6, but I haven't done anything to configure for IPv6 and I'm pretty sure I'd have to configure something on the godaddy end in order to use IPv6 to connect to these hosts.

My system is an Xubuntu 12.04 LTS system where I accept the normal updates. My /var/logs/apt/history.log show updates in the past 2 weeks to openssh-client, openssl, and libssl1. There are others but those are the only ones that look relevant to me.

Uname reports:

Linux Popwease 3.2.0-60-generic #91-Ubuntu SMP Wed Feb 19 03:54:44 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I've been googling and man page searching since last night with very little enlightenment on what's going on.


24 Apr 2014

Some additional information. More evidence of a getaddrinfo issue. I learned that host and dig use the older gethostbyname but that "getent hosts ---" uses getaddrinfo. This explains why I could get IP address from name by using host.

Here's what I get when I use getent:

ccc@Popwease:~$ getent hosts amazon.com
::ffff:67.215.65.145 amazon.com.nycap.rr.com
ccc@Popwease:~$ getent hosts amazon.com.
176.32.98.166   amazon.com
205.251.242.54  amazon.com
72.21.194.212   amazon.com
72.21.215.232   amazon.com
ccc@Popwease:~$ getent hosts gutenberg.org
::ffff:67.215.65.145 gutenberg.org.nycap.rr.com
ccc@Popwease:~$ getent hosts gutenberg.org.
152.19.134.47   gutenberg.org
ccc@Popwease:~$ getent hosts www.gutenberg.org
152.19.134.47   gutenberg.org www.gutenberg.org
ccc@Popwease:~$ getent hosts www.amazon.com
::ffff:67.215.65.145 www.amazon.com.nycap.rr.com

Note that ".nycap.rr.com" is the domain of my internet provider. So unless I put a final '.' on the domain name, getent returns a bogus IPv6 address. But www.gutenberg.org seems to be an exception. OK, amazon.com is a domain and you're relying on a DNS default to get a host. But the final example seems to cloud that theory.

The webhost that I was trying to get to originally uses a defaulted host for the domain,but does have a 'www'. When I either put a final dot on the name or give the full name, I no longer get the hanging on the programs concerned. It looked to me like the godaddy DB server was a full name, but it requires a final '.' also to work.

2 Answers 2

0

I am answering this about 1 hour after I made the last edit to the original post. Without any change on my system, getent started working correctly as well as the programs that originally prompted this post.

I must conclude that the problem was an error on the DNS server at roadrunner which has just been corrected.

0

I had a similar problem. My CentOS 5.5 OpenVZ server (2.6.18-238.5.1.el5.028stab085.5) would hang many programs, like ssh or even sudo and they all stopped on what seems like a network call:

socket(PF_NETLINK, SOCK_RAW, 0)         = 5
bind(5, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(5, {sa_family=AF_NETLINK, pid=115051, groups=00000000}, [1443221608318631948]) = 0
sendto(5, "\24\0\0\0\22\0\1\3\0345vS\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20

I could not even restart network services. It took a reboot to get it back to normal... anyone understand what is the remote end of this socket?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .