Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've spent half a day trying to figure out why a simple network comms app runs slower on what seems to be a better specced machine.

There are two programs: a simple client which, over TCP/IP connects to the server, writes (the same) message and waits for a reply, before writing the same message again. The simple server waits for the client to write a message, and then writes (the same) message to the client. This exchange is done in a loop 1M times. This test is being done over localhost.

This interaction on the older computer achieves about 90000 messages per second, and on the newer computer achieves about 44000 messages per second, even though the newer computer has better specs, and doesn't have any other user processes running on it.

The older computer:

  • is a shared dev box - various other processes run in tandem with my tests
  • has 16 Intel(R) Xeon(R) CPU X5570 @ 2.93GHz processors
  • has 49Gb of memory, of which only about 5Gb is free

The newer computer:

  • is not shared - the only processes running are my own
  • has 24 Intel(R) Xeon(R) CPU X5690 @ 3.47GHz processors
  • has 24.5Gb of memory, of which 21Gb is free

I have checked the following:

  • OS Kernels are the same. Uname -a shows: Linux ficsydapd20 2.6.18-238.9.1.el5 #1 SMP Fri Mar 18 12:42:39 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
  • memory architecture is better on the newer machine - benchmarked using cachebench
  • newer machine outperformed on a cpu benchmark - Fhourstone
  • strace shows roughly the same amount of system calls
  • vmstat shows no paging while program is running
  • MTU is the same for loopback device via /sbin/ifconfig
  • Default socket buffers used for both systems checked using /proc/sys/net/ipv4/tcp_wmem and /proc/sys/net/ipv4/tcp_rmem
  • localhost is defined as 127.0.0.1 on /etc/hosts for both machines

Does anyone know what could be the issue, or what might provide more information?

share|improve this question
    
The new machine doesn't have a slower NIC? Is it connected to a different switch? –  Robert Jan 6 '12 at 8:39
    
As I'm sending traffic through the loopback device, I wouldn't think the NIC/switch would matter. –  Taras Jan 10 '12 at 0:52
add comment

2 Answers

You need to check a few more things here

1) check that qdisc is not configured.. make sure there is no entry for lo device here

[qdisc is used to rate-limit interfaces]

root@x:~# tc qdisc
qdisc pfifo_fast 0: dev eth0 root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc mq 0: dev wlan0 root 

2) check iptables : make sure thse are same for both machines [ connection logging in iptables or masquerading can reduce effective speed]

root@x:~# iptables -L
root@x:~# iptables -L -t nat

if not you can try using iptables -F  and iptables -t nat -F  to flush them

3) check status of status of selinux is same on both (enabled or disabled)

4) in /proc/sys/net/ipv4 tcp connections rate depends on many parameters like (since you are sending connections at very fast rate)

net.ipv4.tcp_tw_reuse

net.ipv4.tcp_max_syn_backlog

so you will have to ensure that these values are same for both machines

Easy way to spot differences is

Machine 1

root@x:~# sysctl -a | grep net.ipv4.tcp_ > s1

Machine 2

root@x:~# sysctl -a | grep net.ipv4.tcp_ > s2

copy files and check diff

root@x:~# diff s1 s2

You can change the values that are different.

share|improve this answer
    
Unfortunately your excellent suggestions didn't show anything 1) qdisc output shows no entries for lo: qdisc pfifo_fast 0: dev eth0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 qdisc pfifo_fast 0: dev eth4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 2) iptables are not used on the newer computer 3) se linux is disabled on both machines 4) no diffs found in the output of these commands on both of the machines It feels like we're getting to the end of the list of things to check! Any other suggestions? –  Taras Jan 10 '12 at 0:53
    
I am running out of ideas here,but you might want to check again with "sysctl -a | grep net". Also if you are using fork (multiple processes) check "ulimit -u" on both machines. If these dont work ,It would be easier to debug this with a tcpdump capture. –  daya Jan 10 '12 at 3:25
    
the faster older machine has a whole bunch of net.ipv6 entries that are missing in the newer machine. Also the net.ipv4.udp_mem are different, although that's not too promising seeing that I'm not (AFAIK) using UDP. Not using forks so ulmit won't help here. Maybe time for a tcpdump capture then? –  Taras Jan 10 '12 at 4:43
    
yes a tcpdump will tell you whether connections are initiated at a lesser rate or they are being processed slower. –  daya Jan 10 '12 at 4:45
    
So the sysctl results (ipv6) aren't significant? –  Taras Jan 10 '12 at 4:53
show 2 more comments
up vote 1 down vote accepted

We synced up kernel versions and drivers, which didn't help. It turns out some of the BIOS settings were causing performance to drop. In particular:

The C states and C1E options that reduce clock speed and voltage draw when the box isn't being used to save power - because the newer machine wasn't being utilized as much, it was being drawn into this state, which would decrease performance (counter intuitively).

Performance is now meeting and sometimes exceeding that of the older machine.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.