For the longest time I've had an elusive problem with my network.
I've changed router, access point and made wireless connections wired in search for a performance thief without getting any closer to understanding what's wrong with my setup.
The problem is as vague as "the network feels slow" and no particular symptom of the problem has been persistent enough for me to find its root cause.
My current infrastructure is made up of:
A pfsense virtual machine running on esxi. Currently the only vm running on the host (Proliant ML110, Core 2 Duo) to eliminate the other vm's as performance thieves. The server has two NICs, one for WAN and one for LAN.
Three Procurve 1800/1810 8 and 24 port switches. Two VLANs, one for LAN and one for WAN.
One Ubiquiti Unifi UAP-AC.
That network serves 20-something units with connectivity.
Yesterday there was a more persistant issue where I was unable to start a movie on Netflix, a bit of googling brought me here after having Netflix support tell me the problem is not with them but with my ISP.
The description there matches the problems I have well, starting the app is very slow, playback is not always working.
I tried unplugging the Apple-TV and connecting my Macbook using the same cable. On the computer Netflix worked without a hitch. Running a speedtest on that very cable I verified my bandwidth to 100 bidirectional megabits. Reconnecting the Apple-TV Netflix refused to work.
Changing the vlan on the port to which the Apple-TV is connected from the LAN to the WAN made it connect directly to the internet with a public IP, with which it could stream the movie without problem.
Putting it back on the LAN playback yet again failed. I disconnected everything but the Apple-TV and the switch uplink from the switch. Went to the switch with the internet incoming, disconnected everything but the two pfsense ports, the connection to the fibre converter and the uplink to the switch with the Apple-TV. After which it was able to start playback.
Consequentially I reasoned that I should be able to pinpoint the troublemaker by reconnecting everything and see which cable kills the playback. None did. Everything went on working when everything was connected again.
Such has been my experience everytime I try to figure out why my connection performs poorly. On 100 bidirectional megabits it ought to be rather snappy, but I've several times switched wifi off on my cell phone because 4G is faster. Speedtest always shows 100 megabits.
Streaming seems particularly hard to handle, mirroring my screen using airplay is practically useless. Playing music using the same technology works but playback disruptions are common.
While yesterday bypassing the firewall seemed to indicate that it is the crook in all this, today the results are reversed:
$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..100}; do ping -c 1 -s 1600 -M dont google.se > /dev/null; done
inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0
real 2m23.383s
user 0m0.046s
sys 0m0.253s
$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..100}; do ping -c 1 -s 1600 -M dont google.se > /dev/null; done
inet 10.11.12.162/24 brd 10.11.12.255 scope global eth0
real 0m52.497s
user 0m0.054s
sys 0m0.253s
Also, to try to verify the MTU for the network (per the apple forum theory) I tried pinging with different package sizes from the WAN, my tests showing that big packages did not traverse the network well:
$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..2}; do ping -c 1 -s 1500 google.se ; done
inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0
PING google.se (64.233.163.94) 1500(1528) bytes of data.
--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING google.se (64.233.163.94) 1500(1528) bytes of data.
--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
real 0m20.015s
user 0m0.003s
sys 0m0.003s
Some experimentation seemed to verify that the MTU is indeed 1500 on the WAN:
$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..2}; do ping -c 1 -s 1472 google.se ; done
inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0
PING google.se (216.58.209.131) 1472(1500) bytes of data.
72 bytes from arn09s05-in-f3.1e100.net (216.58.209.131): icmp_seq=1 ttl=59 (truncated)
--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.497/4.497/4.497/0.000 ms
PING google.se (216.58.209.131) 1472(1500) bytes of data.
72 bytes from arn09s05-in-f3.1e100.net (216.58.209.131): icmp_seq=1 ttl=59 (truncated)
--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.454/4.454/4.454/0.000 ms
real 0m0.034s
user 0m0.003s
sys 0m0.003s
$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..2}; do ping -c 1 -s 1473 google.se ; done
inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0
PING google.se (216.58.209.131) 1473(1501) bytes of data.
--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING google.se (216.58.209.131) 1473(1501) bytes of data.
--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
real 0m20.018s
user 0m0.001s
sys 0m0.007s
On the LAN the breakpoint is on the same packet size, but I must manually specify that I don't allow fragmentation for the packet to break:
$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 68:b5:99:e7:07:a8 brd ff:ff:ff:ff:ff:ff
inet 10.11.12.162/24 brd 10.11.12.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::6ab5:99ff:fee7:7a8/64 scope link
valid_lft forever preferred_lft forever
$ ping -c 1 -s 3000 google.se
PING google.se (83.255.235.35) 3000(3028) bytes of data.
3008 bytes from 83.255.235.35: icmp_seq=1 ttl=61 time=82.3 ms
--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 82.324/82.324/82.324/0.000 ms
$ ping -c 1 -s 1472 -M do google.se
PING google.se (83.255.235.123) 1472(1500) bytes of data.
1480 bytes from cache.google.com (83.255.235.123): icmp_seq=1 ttl=61 time=74.7 ms
--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 74.779/74.779/74.779/0.000 ms
$ ping -c 1 -s 1473 -M do google.se
PING google.se (83.255.235.35) 1473(1501) bytes of data.
ping: local error: Message too long, mtu=1500
--- google.se ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
I don't understand what's wrong with my network and I can't figure out how to troubleshoot it further. Please help me restructure my troubleshooting so that I can rid the network of ghosts once and for all.
EDIT, Regarding my DNS settings:
If I understand that correctly it means that googles resolvers are only used as a fallback when the DHCP assigned name server from the ISP is unavailable. Is that correct? Or am I randomly asking a name server far far away for adresses?
As you can see below pfsense tries to handle name resolution by itself first, asking the ISP second and thirdly and only resorting to google as a fourth and fifth option, which seems quite reasonable to me?
The Apple TV has DHCP assigned network settings, and uses the gateway as name server. The DHCP server has no dedicated dns settings, but inherits the name server list above:
EDIT, Regarding the for loop:
The reason for running ping 100 times rather than once with 100 packets is/was to have name resolution performed each time as it seemed to take quite varying amounts of time to get ping "started" when I ran it by hand and I thought maybe I could make that feeling more distinct by multiplying the action by 100.
Given that Ubuntu has the following config:
$ grep nameserver /etc/resolv.conf
nameserver 127.0.1.1
That thought might be a little silly tough...
EDIT, Regarding the Apple TV:
I have reset the Apple TV to factory defaults. I have unplugged the device on multiple occasions (sometimes with not so little fury in the blood).
EDIT, pfSense:
I factory defaulted the pfSense the other day and only re-enabled the more essential pieces of it (dns, dhcp, nat, static dhcp-leases, a few port forwardings), but yesterday Netflix still stopped playing mid stream. The problem disappeared again though, so re-starting the movie worked.
I wonder if Netflix needs DNS mid stream, feels like that should already been taken care of by then.
And since the server is running the latest pfsense version (2.2.2) it uses unbound by default.
That the resolver is successfully caching resolutions can be verified by doing two consecutive lookups with the diagnostics tool:
The different answers confuse me though.
EDIT, MTU:
The MTU is set to automatic.
EDIT, ATV speed test:
Doing a speed test on the Apple TV produces the following graph on the pfsense:
After which the Apple TV says "test finished successfully", but only for a split second, then it changes to:
I can use Netflix despite the error.
ping -c 100 …
. In fact, why don't you do aping -c 100 …
and post the results to a pastebin or github gist and link to it from here? It would be interesting to see the packet loss rates and latency variations.