1

For the longest time I've had an elusive problem with my network.

I've changed router, access point and made wireless connections wired in search for a performance thief without getting any closer to understanding what's wrong with my setup.

The problem is as vague as "the network feels slow" and no particular symptom of the problem has been persistent enough for me to find its root cause.

My current infrastructure is made up of:

A pfsense virtual machine running on esxi. Currently the only vm running on the host (Proliant ML110, Core 2 Duo) to eliminate the other vm's as performance thieves. The server has two NICs, one for WAN and one for LAN.

Three Procurve 1800/1810 8 and 24 port switches. Two VLANs, one for LAN and one for WAN.

One Ubiquiti Unifi UAP-AC.

That network serves 20-something units with connectivity.

Yesterday there was a more persistant issue where I was unable to start a movie on Netflix, a bit of googling brought me here after having Netflix support tell me the problem is not with them but with my ISP.

The description there matches the problems I have well, starting the app is very slow, playback is not always working.

I tried unplugging the Apple-TV and connecting my Macbook using the same cable. On the computer Netflix worked without a hitch. Running a speedtest on that very cable I verified my bandwidth to 100 bidirectional megabits. Reconnecting the Apple-TV Netflix refused to work.

Changing the vlan on the port to which the Apple-TV is connected from the LAN to the WAN made it connect directly to the internet with a public IP, with which it could stream the movie without problem.

Putting it back on the LAN playback yet again failed. I disconnected everything but the Apple-TV and the switch uplink from the switch. Went to the switch with the internet incoming, disconnected everything but the two pfsense ports, the connection to the fibre converter and the uplink to the switch with the Apple-TV. After which it was able to start playback.

Consequentially I reasoned that I should be able to pinpoint the troublemaker by reconnecting everything and see which cable kills the playback. None did. Everything went on working when everything was connected again.

Such has been my experience everytime I try to figure out why my connection performs poorly. On 100 bidirectional megabits it ought to be rather snappy, but I've several times switched wifi off on my cell phone because 4G is faster. Speedtest always shows 100 megabits.

Streaming seems particularly hard to handle, mirroring my screen using airplay is practically useless. Playing music using the same technology works but playback disruptions are common.

While yesterday bypassing the firewall seemed to indicate that it is the crook in all this, today the results are reversed:

$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..100}; do ping -c 1 -s 1600 -M dont google.se > /dev/null; done
    inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0

real        2m23.383s
user        0m0.046s
sys 0m0.253s
$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..100}; do ping -c 1 -s 1600 -M dont google.se > /dev/null; done
    inet 10.11.12.162/24 brd 10.11.12.255 scope global eth0

real        0m52.497s
user        0m0.054s
sys 0m0.253s

Also, to try to verify the MTU for the network (per the apple forum theory) I tried pinging with different package sizes from the WAN, my tests showing that big packages did not traverse the network well:

$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..2}; do ping -c 1 -s 1500 google.se ; done
    inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0
PING google.se (64.233.163.94) 1500(1528) bytes of data.

--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING google.se (64.233.163.94) 1500(1528) bytes of data.

--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms


real        0m20.015s
user        0m0.003s
sys 0m0.003s

Some experimentation seemed to verify that the MTU is indeed 1500 on the WAN:

$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..2}; do ping -c 1 -s 1472 google.se ; done
    inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0
PING google.se (216.58.209.131) 1472(1500) bytes of data.
72 bytes from arn09s05-in-f3.1e100.net (216.58.209.131): icmp_seq=1 ttl=59 (truncated)

--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.497/4.497/4.497/0.000 ms
PING google.se (216.58.209.131) 1472(1500) bytes of data.
72 bytes from arn09s05-in-f3.1e100.net (216.58.209.131): icmp_seq=1 ttl=59 (truncated)

--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.454/4.454/4.454/0.000 ms

real        0m0.034s
user        0m0.003s
sys 0m0.003s
$ ip addr show dev eth0 | grep "inet\b" && time for i in {1..2}; do ping -c 1 -s 1473 google.se ; done
    inet 80.216.153.211/22 brd 80.216.155.255 scope global eth0
PING google.se (216.58.209.131) 1473(1501) bytes of data.

--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING google.se (216.58.209.131) 1473(1501) bytes of data.

--- google.se ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms


real        0m20.018s
user        0m0.001s
sys 0m0.007s

On the LAN the breakpoint is on the same packet size, but I must manually specify that I don't allow fragmentation for the packet to break:

$ ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 68:b5:99:e7:07:a8 brd ff:ff:ff:ff:ff:ff
    inet 10.11.12.162/24 brd 10.11.12.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::6ab5:99ff:fee7:7a8/64 scope link
       valid_lft forever preferred_lft forever

$ ping -c 1 -s 3000 google.se
PING google.se (83.255.235.35) 3000(3028) bytes of data.
3008 bytes from 83.255.235.35: icmp_seq=1 ttl=61 time=82.3 ms

--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 82.324/82.324/82.324/0.000 ms

$ ping -c 1 -s 1472 -M do google.se
PING google.se (83.255.235.123) 1472(1500) bytes of data.
1480 bytes from cache.google.com (83.255.235.123): icmp_seq=1 ttl=61 time=74.7 ms

--- google.se ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 74.779/74.779/74.779/0.000 ms

$ ping -c 1 -s 1473 -M do google.se
PING google.se (83.255.235.35) 1473(1501) bytes of data.
ping: local error: Message too long, mtu=1500

--- google.se ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

I don't understand what's wrong with my network and I can't figure out how to troubleshoot it further. Please help me restructure my troubleshooting so that I can rid the network of ghosts once and for all.

EDIT, Regarding my DNS settings:

pfsense dns settings

If I understand that correctly it means that googles resolvers are only used as a fallback when the DHCP assigned name server from the ISP is unavailable. Is that correct? Or am I randomly asking a name server far far away for adresses?

As you can see below pfsense tries to handle name resolution by itself first, asking the ISP second and thirdly and only resorting to google as a fourth and fifth option, which seems quite reasonable to me?

pfsense DNS server list

The Apple TV has DHCP assigned network settings, and uses the gateway as name server. The DHCP server has no dedicated dns settings, but inherits the name server list above:

enter image description here

EDIT, Regarding the for loop:

The reason for running ping 100 times rather than once with 100 packets is/was to have name resolution performed each time as it seemed to take quite varying amounts of time to get ping "started" when I ran it by hand and I thought maybe I could make that feeling more distinct by multiplying the action by 100.

Given that Ubuntu has the following config:

$ grep nameserver /etc/resolv.conf 
nameserver 127.0.1.1

That thought might be a little silly tough...

EDIT, Regarding the Apple TV:

I have reset the Apple TV to factory defaults. I have unplugged the device on multiple occasions (sometimes with not so little fury in the blood).

EDIT, pfSense:

I factory defaulted the pfSense the other day and only re-enabled the more essential pieces of it (dns, dhcp, nat, static dhcp-leases, a few port forwardings), but yesterday Netflix still stopped playing mid stream. The problem disappeared again though, so re-starting the movie worked.

I wonder if Netflix needs DNS mid stream, feels like that should already been taken care of by then.

And since the server is running the latest pfsense version (2.2.2) it uses unbound by default.

That the resolver is successfully caching resolutions can be verified by doing two consecutive lookups with the diagnostics tool:

first second

The different answers confuse me though.

EDIT, MTU:

The MTU is set to automatic.

MTU

EDIT, ATV speed test:

Doing a speed test on the Apple TV produces the following graph on the pfsense: graph

After which the Apple TV says "test finished successfully", but only for a split second, then it changes to:

error

I can use Netflix despite the error.

  • What DNS server addresses are your Apple TV using? – Spiff May 12 '15 at 20:31
  • I've never seen someone do 100 separate single-count ping invocations and look at the total time taken. Seems like that would provide far less useful information than doing a ping -c 100 …. In fact, why don't you do a ping -c 100 … and post the results to a pastebin or github gist and link to it from here? It would be interesting to see the packet loss rates and latency variations. – Spiff May 13 '15 at 0:39
  • Have you tried removing the Apple TV from your Netflix devices list and re-adding it? Have you unplugged the Apple TV for 5 minutes and plugged it back in again? Next time you have issues, fire up wireshark and see if there's anything freaky going on. Set pfsense to auto select the MTU. If no, use 1500, or 1518 if that doesn't work. – Alex Atkinson May 13 '15 at 1:24
  • @azzid I'm not familiar enough with pfsense and your network topology to make sense of those screenshots. I see pfsense has a package called "Unbound" that can either act as a resolving DNS proxy or as a simple DNS forwarder. How exactly do you have that configured? You have something on the pfsense box set to get DNS addresses from DHCP/PPP on WAN. Could that be coming into play here? Also, do you know for a fact that pfsense only uses the first DNS entry unless it fails, and only then moves down the list? Or might it be using the whole list all the time in round-robin fashion? – Spiff May 18 '15 at 18:13
  • @assid That said, my best advice to you is to make sure you're always using your own ISP's most local DNS servers and never anyone else's. If I were you, I'd do a test by removing 8.8.8.8 and 8.8.4.4 from everywhere, and see if the problem goes away. Or maybe by temporarily giving my Apple TV a static IP configuration so I can set its DNS servers manually, and setting them to only the local ISP's (comhem.se?) DNS servers, not Google and not the pfsense box (since it's unclear exactly what the pfsense box is doing). – Spiff May 18 '15 at 18:14
2

Make sure to use your local ISP's DNS server, not some distant DNS service.

Most of the content you can download or stream to your Apple TV comes via the Akamai CDN (Apple's been one of Akamai's biggest customers for ages). Akamai finds the CDN edge node (server) nearest you based on where your DNS lookup comes from, and your DNS lookup generally comes from whichever local recursive/resolving DNS server you've configured your client device to use.

Make sure your Apple TV is set to use your local ISP's DNS servers, and not some potentially distant servers like Google DNS (8.8.8.8 and 8.8.4.4) or Level 3 (4.2.2.x) or OpenDNS or anyone like that. Note that your Apple TV may be getting its DNS settings via DHCP, and your DHCP server may be a process on your router/gateway. If the DHCP server is telling your Apple TV to use the private IP address of your NAT gateway (or other local firewall or router) as its DNS address, it means your NAT gateway is acting as a DNS proxy. If you want to keep doing that, make sure that that NAT gateway is using your local ISP's DNS servers as its DNS servers.

By using a local DNS server, Akamai will direct your client to make download/stream requests from your nearest Akamai server in Sweden, as opposed to, say, some server near a Google data center in the USA where Google's 8.8.8.8 DNS server is located.

[Even if this isn't the right answer for you, it might be the right answer for other people finding this question, so I'm going to leave it here anyway.]

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.