Why does a Bidirectional iperf request freezes on the server to client side?

Question

To diagnose the difference in network trafic between two locations and our main server center, I'm using iperf to log the traffic rate with the following command:

iperf -c <server ip> -F 500MB_Test_File.zip -n 500M -y C

After noting disparities between upload and download rate, I wanted to use the -d option and stumbled on a weird behaviour: although the transfer went through in the client to server direction, in the other direction it freezed and didn't seem to tiemout. Here is a test I did which shows that it isn't connected to the fact I'm using a file as source for the data transfer:

$ iperf -c <server IP> -d -i 1
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to <server ip>, TCP port 5001
TCP window size:  153 KByte (default)
------------------------------------------------------------
[  5] local <local ip> port 35144 connected with <server ip> port 5001
[  4] local <local ip> port 5001 connected with <server ip> port 39967
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 1.0 sec  9.75 MBytes  81.8 Mbits/sec
[  4]  0.0- 1.0 sec  10.7 MBytes  89.3 Mbits/sec
[  5]  1.0- 2.0 sec  10.9 MBytes  91.2 Mbits/sec
[  4]  1.0- 2.0 sec  10.9 MBytes  91.3 Mbits/sec
[  5]  2.0- 3.0 sec  11.0 MBytes  92.3 Mbits/sec
[  4]  2.0- 3.0 sec  10.9 MBytes  91.4 Mbits/sec
[  5]  3.0- 4.0 sec  11.0 MBytes  92.3 Mbits/sec
[  4]  3.0- 4.0 sec  10.9 MBytes  91.4 Mbits/sec
[  5]  4.0- 5.0 sec  11.0 MBytes  92.3 Mbits/sec
[  4]  4.0- 5.0 sec  10.9 MBytes  91.4 Mbits/sec
[  5]  5.0- 6.0 sec  11.0 MBytes  92.3 Mbits/sec
[  4]  5.0- 6.0 sec  10.9 MBytes  91.4 Mbits/sec
[  5]  6.0- 7.0 sec  10.9 MBytes  91.2 Mbits/sec
[  4]  6.0- 7.0 sec  10.9 MBytes  91.4 Mbits/sec
[  5]  7.0- 8.0 sec  11.2 MBytes  94.4 Mbits/sec
[  4]  7.0- 8.0 sec  10.9 MBytes  91.4 Mbits/sec
[  5]  8.0- 9.0 sec  11.1 MBytes  93.3 Mbits/sec
[  5]  9.0-10.0 sec  11.1 MBytes  93.3 Mbits/sec
[  5]  0.0-10.0 sec   109 MBytes  91.4 Mbits/sec
^C^C

I'm using iperf 2.0.5. Both clients and the server are linux machines, server being virtual.

EDIT

After following MariusMatutiae's suggestion I started the iperf server on the client machine and then tried to make the bidirectional request from the VM and here is what I got:

[host@centos ~]$ iperf -c <client_id> -d -i 1
bind failed: Address already in use
------------------------------------------------------------
Client connecting to <client_id>, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[  4] local <server_id> port 40175 connected with <client_id> port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  11.6 MBytes  97.5 Mbits/sec
[  4]  1.0- 2.0 sec  9.50 MBytes  79.7 Mbits/sec
[  4]  2.0- 3.0 sec  10.6 MBytes  89.1 Mbits/sec
[  4]  3.0- 4.0 sec  11.0 MBytes  92.3 Mbits/sec
[  4]  4.0- 5.0 sec  11.0 MBytes  92.3 Mbits/sec
[  4]  5.0- 6.0 sec  10.9 MBytes  91.2 Mbits/sec
[  4]  6.0- 7.0 sec  11.0 MBytes  92.3 Mbits/sec
[  4]  7.0- 8.0 sec  10.8 MBytes  90.2 Mbits/sec
[  4]  8.0- 9.0 sec  8.38 MBytes  70.3 Mbits/sec
[  4]  9.0-10.0 sec  9.75 MBytes  81.8 Mbits/sec
[  4]  0.0-10.0 sec   105 MBytes  87.8 Mbits/sec
[host@centos ~]$

I thought, great, the command stops from itself, but I only get the results in one direction. That is until I looked into the machine where I started the iPerf server this time and saw this:

$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local <client_ip> port 5001 connected with <server_ip> port 40179
------------------------------------------------------------
Client connecting to <server_ip>, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  6] local <client_ip> port 36243 connected with <server_ip> port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  45.4 MBytes  38.0 Mbits/sec
[  4]  0.0-10.2 sec  64.2 MBytes  52.8 Mbits/sec

So it got me thinking that maybe I need to start a server as daemon on both side of the connection for the bidirectional command to work and indeed, I got a similar results, only outputing the performance in one side of the transaction, but as such I don't have any way to get the performance data in the other side: I did start the iperf daemon with -D > iperflog but the log doesn't seem to be populated...

Is it suppose to be like that, that a iperf is supposed to be running in both machine for a bidirectional request to go through? Does the bind failed: Address already in use message i get when I start an iperf client and a server is running on both side a clue as where the issue might lie? I get the feeling that what I am doing is not supposed to be the standard procedure...

Answer 1

As I suspected and Marius confirmed client and server can't run on the same machine. So I waited for an opportune time to kill the daemon so it wouldn't run concurrently to the client, started the server only with -s on the host (which is on a totally other location, as Marius asked), started the bidirectional request on the VM and the request went through (Thought of the moment: "What the...").

And then it hit me, I get the issue only if the server is started as daemon.

I confirmed my hypothesis by testing both with server as a daemon and not a daemon, as well as swapping the role of the machines.

Summary: If you want to use bidirectional requests, don't start the server as a daemon.