Performance decrease on simple network app when run on 'better' machine

Question

I've spent half a day trying to figure out why a simple network comms app runs slower on what seems to be a better specced machine.

There are two programs: a simple client which, over TCP/IP connects to the server, writes (the same) message and waits for a reply, before writing the same message again. The simple server waits for the client to write a message, and then writes (the same) message to the client. This exchange is done in a loop 1M times. This test is being done over localhost.

This interaction on the older computer achieves about 90000 messages per second, and on the newer computer achieves about 44000 messages per second, even though the newer computer has better specs, and doesn't have any other user processes running on it.

The older computer:

• is a shared dev box - various other processes run in tandem with my tests
• has 16 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz processors
• has 49Gb of memory, of which only about 5Gb is free

The newer computer:

• is not shared - the only processes running are my own
• has 24 Intel(R) Xeon(R) CPU X5690 @ 3.47GHz processors
• has 24.5Gb of memory, of which 21Gb is free

I have checked the following:

• OS Kernels are the same. Uname -a shows: Linux ficsydapd20 2.6.18-238.9.1.el5 #1 SMP Fri Mar 18 12:42:39 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
• memory architecture is better on the newer machine - benchmarked using cachebench
• newer machine outperformed on a cpu benchmark - Fhourstone
• strace shows roughly the same amount of system calls
• vmstat shows no paging while program is running
• MTU is the same for loopback device via /sbin/ifconfig
• Default socket buffers used for both systems checked using /proc/sys/net/ipv4/tcp_wmem and /proc/sys/net/ipv4/tcp_rmem
• localhost is defined as 127.0.0.1 on /etc/hosts for both machines

Does anyone know what could be the issue, or what might provide more information?

Answer 1

You need to check a few more things here

1) check that qdisc is not configured.. make sure there is no entry for lo device here

[qdisc is used to rate-limit interfaces]

root@x:~# tc qdisc
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc mq 0: dev wlan0 root 

2) check iptables : make sure thse are same for both machines [ connection logging in iptables or masquerading can reduce effective speed]

root@x:~# iptables -L
root@x:~# iptables -L -t nat

if not you can try using iptables -F  and iptables -t nat -F  to flush them

3) check status of status of selinux is same on both (enabled or disabled)

4) in /proc/sys/net/ipv4 tcp connections rate depends on many parameters like (since you are sending connections at very fast rate)

net.ipv4.tcp_tw_reuse

net.ipv4.tcp_max_syn_backlog

so you will have to ensure that these values are same for both machines

Easy way to spot differences is

Machine 1

root@x:~# sysctl -a | grep net.ipv4.tcp_ > s1

Machine 2

root@x:~# sysctl -a | grep net.ipv4.tcp_ > s2

copy files and check diff

root@x:~# diff s1 s2

You can change the values that are different.

Answer 2

We synced up kernel versions and drivers, which didn't help. It turns out some of the BIOS settings were causing performance to drop. In particular:

The C states and C1E options that reduce clock speed and voltage draw when the box isn't being used to save power - because the newer machine wasn't being utilized as much, it was being drawn into this state, which would decrease performance (counter intuitively).

Performance is now meeting and sometimes exceeding that of the older machine.