0

I have been seeing a weird problem that I can't reproduce on demand and have a suspicion on the root cause.

The Issue: Intermittently the entire network goes down until I walk around and unplug whatever computer is causing the spanning tree flood.

Topology: I've got 2 Cisco unmanaged gigabit switches connected via gigabit gbic. Both switches have the corresponding port next to the gigabit gbic port unoccupied so that the up-link functions as designed. Both Switches are Cisco and the same family (SG100 and SG102) so it's not a matter of incompatibility.

I've taken a wireshark capture directly connected to the culprit machine as well as connected via the switch and BOTH yield the same spanning Tree flood causing the MAC PAUSE frame to slow stuff down which kills the network.

Probable culprit but unable to replicate issue "YET" is that this seems to usually occur AFTER the following occurs:
1. User undocks their laptop from their docking station and connects to WiFi
2. User is done with need for laptop away from desk and re-docks
3. User's laptop re-connects via Ethernet on the docking station
4. Sometimes crashes entire network.

Since I've been unable to replicate the issue on demand, how could I build a filter of some sort for Wireshark to capture only the packets that would resemble an echo (NOT ICMP ECHO) more like duplicated traffic causing the initial storm before it goes nuts with spanning tree?

This way I could run the capture for days or weeks until it occurs again. Below is what I'm seeing after the network goes down in wireshark.

Since these are not managed switches they don't even support STP so I'm boggled on why it always ends with spanning tree traffic. Also the source MAC address does not exist in a natural configuration and I only know the affected workstation after the fact and it also is always frozen or occasionally gotten a BSOD. It's been a LONG time since I've seen a BSOD when this happens but the system frozen occurs every time and there is no minidump and yes it's configured.

Other things I've already eliminated:
Cabling or cabling loop(s)
event logs - just show time loss between frozen time and reboot
no dumps when frozen
updated to Dell's latest certified drivers and BIOS
rebooted everything (again intermittent but usually after a undock, connecft to WiFi, re-dock and auto connect to ethernet pattern)

enter image description here

1 Answer 1

1

First, just to be clear, this is not Spanning Tree Protocol (IEEE 802.1D), this is Ethernet flow control (IEEE 802.3x, now part of IEEE 802.3-2012). Ethernet flow control PAUSE frames are addressed to one of the same addresses that STP uses, so packet sniffers often report the address as an STP address even when it's being used for flow control.

The 802.3x era of Ethernet flow control was kind of a bust. It was discovered too late that it could cause problems on the network, especially "head-of-line" blocking. Imagine a fast server serving data to a normal client and a slow client. The slow client gets overwhelmed sends a PAUSE frame to the switch, and now the switch can't deliver all the frames it's getting from the server so the switch sends a PAUSE frame to the server. This blocks the server from being able to send frames to the other (normal) client, even though the server, switch, and client all have the spare capacity for it. This one slow client (and a none-too-bright switch and the none-too-bright 802.3x Ethernet flow control protocol) screwed things up for everyone.

Because of this, some switch vendors intentionally don't support 802.3x-style flow control, or if they support it at all, they only let the switch honor incoming PAUSE frames, but never send them. If your switches are manageable at all, and have configuration settings for flow control, make sure they're configured to never send PAUSE frames.

In fact, given that you're seeing PAUSE frame floods, your network would probably be better off if you disabled flow control all together. Configure your switches and clients to disable flow control.

Also, keep your Ethernet drivers up to date and consider purging your network of any models of Ethernet NICs that have been known to spam the network with PAUSE frames when the host crashes.

5
  • It makes sense, and I have been able to on one occasion replicate the issue, however it occurred after I finished taking a capture so again the initial cause was missed. I was able to bring down the network after taking a client and ejecting from the laptop dock, connecting WiFi, re-docking and repeating a handful of times. I tried doing this again while I was taking a capture and haven't been able to catch it in the act. All systems are i7 Sandybridge Extreme procs or newer 16-32GB ram and Samsung 840 Pro SSD's and the server is mid-range but there were only 2 people here today when I tested.
    – Brad
    May 27, 2015 at 20:25
  • My best guess is something happening when the laptop is multi-homed on WiFi and Ethernet. I see in DHCP both IP's of a client when I'm docking/Un-docking and connecting to WiFi and hopefully something becomes more visible in wireshark. Also when the PAUSE Frames go crazy whatever machine it's originating from is frozen, but nothing is logged on the system when I cold boot it back up. No logs or dumps so I have nothing else to look at other than the symptom of the network going down.
    – Brad
    May 27, 2015 at 20:30
  • @Brad You might not find an initial cause in a packet trace. It may not have to do with being dual-homed. It may be something like: 1) Something possibly completely unrelated to networking (but that happens during dock/undock) makes Windows die (freeze/BSOD). 2) Because Windows is dead, Ethernet driver is no longer reading frames from Ethernet chip, so Ethernet chip's buffers don't drain, so Ethernet chip starts spamming PAUSE frames for every incoming frame it sees. 3) Buggy switches don't filter PAUSE frames, spam them to all ports.
    – Spiff
    May 27, 2015 at 20:34
  • Makes sense, the switches are unmanaged but for the Ethernet chip to start broadcasting PAUSE wouldn't it need some type of OS or driver logic to say the buffer is full so slow down?
    – Brad
    May 27, 2015 at 20:38
  • @Brad Ethernet chips nowadays contain little microcontrollers and run microcode. The PAUSE frames are almost certainly created and sent at that hardware level.
    – Spiff
    May 27, 2015 at 23:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .