My D-Link's Ethernet bridge downlink just got 10-30x slower?

Question

TL;DR: I unplugged my network to move my desk, and now downloading via my DIR-655's Ethernet LAN bridge is 10-30x slower than the Ethernet switch it's plugged into.

Background

My network is

SMC cable modem <-> Cisco firewall <-> Netgear switch <-> D-Link WiFi†
     |                    |                 |                  |
 SMC8014               ASA-5505         GS608v2 gigE      DIR-655 rev A3 gigE

†The DIR-655 is used as an access point, not a router (although what D-Link calls an access point, I'd call a bridge). The "WAN" port is unused; the Netgear connects to the built-in 4-port Ethernet LAN switch, inside the built- in router/firewall.

Endpoints:

• MacBook Pro 17" mid-2010
• iPhone 4S
• Fedora 12 Linux server running reasonably fast dual-Athlon X2, VelociRaptors, etc.

All cables are <10 feet, mostly CAT-5e, some CAT-6, all premade. All WiFi endpoints are within three feet of the D-Link.

Yesterday I unplugged and rearranged stuff, and now connecting via the D-Link - even through the wired switch, right next to the incoming network cable - is 30x slower than connecting directly to the Netgear switch, on both my MacBook and iPhone.

How I'm measuring "slower"

I'm mostly using http://speedtest.net, which of course only really measures broadband speeds. I've also installed http://www.speedtest.net/mini.php on my local server, but can't test the iPhone with that.

Results

Speedtest.net, closest server over Comcast business-class:

CONFIG                           | PING (ms)   | DOWN (Mbps)   | UP (Mbps)
Mac    <-> Ethernet <-> Netgear  |  9          | 31.6          |  6.8 
Mac    <-> Ethernet <-> D-Link   |  8          |  4.1          |  6.0
Mac    <-> WiFi     <-> D-Link   |  9          |  1.4          |  2.9
iPhone <-> WiFi     <-> D-Link   | 67          |  0.4          |  1.6

Speedtest Mini on Linux PC:

CONFIG                                         | DOWN (Mbps)   | UP (Mbps)
Mac    <-> Ethernet <-> NetGear                | 97.2          | 76.9
Mac    <-> Ethernet <-> D-Link                 |  8.2          | 24.2
Mac    <-> WiFi     <-> D-Link                 |  1.0          |  8.6

Slow typing in SSH:

Mac    <-> Ethernet <-> Netgear <-> Linux PC: smooth
Mac    <-> Ethernet <-> D-Link  <-> Linux PC: choppy

Note that D-Link upload speeds are normal on broadband, slower locally (but I'd believe that's a D-Link limitation), and always faster than the downloads! Since ssh is choppy just with slow typing, I don't believe it's a throttling-type problem either; that's not a lot of bandwidth.

What I've tried

• Swapping all "good" and "bad" cables
• Re-plugging "bad" cable from D-Link to Netgear and watching it be the "good" cable
• pulling cables away from power lines
• Verify that the Mac auto-detects the D-Link as gigE
• Try to verify the link speed of the D-Link <-> Netgear connection, but the firmware doesn't report that
• Verify that the D-Link sees no TX/RX errors or collisions
• Use different Ethernet ports on both Netgear and D-Link
• Reset the D-Link to factory settings
• Upgrade the D-Link firmware from 1.21 to 1.35NA, 2010/11/12, the latest
• Reboot everything at least once
• On the Mac, disable Wi-Fi during the Ethernet tests, and unplug Ethernet during the Wi-Fi tests
• Using iStumbler, verify that the D-Link isn't picking overloaded Wi-Fi channels (usually just 1-5 neighbors on my and adjacent channels, average for my apt building)
• Verify that the only client connected to the Wi-Fi was the iPhone
• Verify that nothing was being chatty on my network according to the WISH log
• Enable and disable all sorts of D-Link settings, including forcing WAN auto-detect to gigE

So.

I don't mind buying a new access point—I wouldn't mind having a dual-link network—but as a guy who's been networking since gated v4 was a drastic rewrite, and who often used physical sniffers in the days before Wireshark, I'm baffled. I hate being baffled.

What could I possibly have changed that would result in this? How can I measure it? All I can think of is a static zap—thick carpet, socks, HVAC—but I didn't feel one, and does that really happen anymore?

Can I test if it's Ethernet vs. TCP layer slowness? I'm not familiar with modern network utilities; it's hard to Google without hitting "Q: Why is my network slow? A: Is your microwave on?"

If I don't get an answer here, will someone big and powerful help me migrate it to serverfault without getting screamed back here? In the words of Inigo Montoya, "I must know." Don't get all Dread Pirate Roberts on me.

Answer 1

I had a similar issue with a netgear switch that had a "uplink" port that was actually shared between two interfaces. If you had something plugged into the port next to the uplink the speed would drop by an order of magnitude. I too have considerable network experience so I broke out tcpdump and really over-analyzed the hell out of it.

I moved the cable over one port, voila, speed returned. This affected wireless and wired clients. It was the most bizarre thing. I pulled the switch, tossed in a cisco 2548 and never had the problem again.

Unfortunately, I am not sure if they have ethtool or mii-tool or the standard *nix t-shooting tools.

SSH being choppy sounds like a nagles algorithm or slow-start type issue slowly releasing packets into the network. Have you confirmed that there really is no network congestion?

This sounds exactly like the issue I was having.