We recently had a new VoIP phone system installed from Star2Star with brand new Yealink T46S phones, and I've been chasing networking issues down ever since. It started on the day of installation, one of our workstations was unable to access some (but not all) of the shares we have mapped on the computer. Looking at the shares that were having a problem, they were all on the same subnet (192.X.X.X), which happens to be the same subnet the workstation is on. The ones that were working were on a different subnet (172.X.X.X). The PC is connected to the network through the phone's Gigabit connection.
The latter (working) shares are on servers hosted by a 3rd party while the former (failing) shares are located physically on in-house resources used for document storage and such. The devices that are "failing" include a couple of virtual servers and a NAS. The specific issue is that the user can't browse to these resources through File Explorer at all, whether I try by name or by IP address. At times I could get to the "root" of the server (name or IP), but as soon as I tried to get to any of the shares, File Explorer seems to just die. The user could PING the devices by name or IP without issue, so I know that the network itself is working "normally", in general. If I bypassed the phone, the problem would seem to go away, but as soon as I put the phone back inline, the issues returned.
I went through a variety of troubleshooting steps trying to fix it, including (in no particular order):
- Disconnecting/reconnecting the mapped drives (
net use X: \\SERVER\Share) - Resetting the winsock (
netsh winsock reset) - System file check (
sfc /scannow) - Clear DNS cache (
ipconfig /flushdns) - Clear arp cache (
arp -d) - Check Windows Firewall (it's disabled for the local network)
- Verify SMB is enabled (we use SMBv1 for a network scanner/printer device for delivery of documents to each user's local computer)
As well as a few other things, but I can't remember them all right now.
Next, in an attempt to rule out the phone itself as the cause, we brought in a small PoE switch and put that in-line between the user's ethernet outlet and the phone/PC (one port connected to the jack, one to the phone, and one to the workstation). Unfortunately, the problem persisted - unable to browse shares on the local LAN servers while remote servers and all Internet services appear to work just fine. This even persisted when I left the PoE switch there but disconnected the phone. It seems that anything between the PC and the network seems to cause this issue, but only for computers on the same subnet.
We tried a variety of physical connection configurations to try to narrow the problem down further, including:
- Repatching the ethernet jack to a different port on the switch (phone still inline with the PC) - Problem persists
- Running ethernet to another patch panel/switch port ((phone still inline with the PC*) - Problem persists
- Taking the same phone to a different PC (phone inline with the "new" PC) - No problem exhibited
Finally, I tried disabling the on-board NIC (Realtek PCIe GBE Family Controller) and plugging in a little Anker USB NIC I had lying around for testing purposes. Everything started working normally (cue angelic choirs singing). I ordered up and installed a new Intel PCIe Gigabit CT Desktop Adapter to replace the Realtek and all seems to be well with the world... at least for that user.
HOWEVER, the very next day a second PC started exhibiting the exact same behavior - can't get to shares on servers within the same subnet, but every other kind of network browsing goes through like a champ. I went ahead and installed another new Intel NIC (same model) in that computer and it seemed to resolve the issue, but only temporarily. This morning I came in and had the user try to get to the problem resources, and it just "hangs" there and eventually gives the error that it can't access the mapped drive/share. Sometimes it will even lock up Windows Explorer to the point that I have to completely restart the computer just to get everything back to a relatively stable and workable environment.
I'm going through a lot of the same troubleshooting as before with pretty much the same results (see above), but I'm still unable to get the PC to browse the network normally, even on the new NIC. It's even causing some issues with some of our in-house applications that rely on browsing to save documents and such. The applications themselves seem to work fine for the most part.
Before the installation of this new phone system, we had a different but similar system in place using Iwatsu VoIP phones. Those devices had a 10/100 pass-through instead of the Gigabit pass-through of these new Yealink phones, but the basic "idea" is the same. While I'd love to cast all the blame on the new phone system, the fact is that (so far, at least) these are the only two computers on the network that have exhibited this behavior. My own computer doesn't have any of these problems, but that may have something to do with the fact that I have a "custom" setup in my office where the phone and the PC are on different Ethernet connections back to the switch. However, we have seven other computers in the office that have yet to report any new/similar problems since the new phone system's installation.
The only difference between this second computer and the first that comes to mind at this point is that the second PC's IP address is assigned via DHCP reservation (for CFS policies in the firewall) whereas the first PC's IP is just picking up a "random" DHCP address. I did have to do some work in the DHCP server's reservation to get the second PC to pick up the correct IP address due to the new NIC's new MAC address, but the PC is currently picking up the correct IP address.
Because it's currently the middle of the work day, I haven't yet tried uninstalling/reinstalling the "Client for Microsoft Networks" on the problem PC, but that's my next step. Regardless, I could really use some help with any other suggestions for places to look.
Additional Details (Environment)
- The phone system is on a separate VLAN (
10.X.X.X). - The phones themselves are getting IP addresses from that VLAN while the workstations are getting appropriate IP addresses from our "main" subnet (
192.X.X.X). - We have other VLAN's configured in the network that appear to be working normally.
- Sonicwall firewall device on the edge of our network.
- 2 UniFi 24-port managed PoE switches and 2 Cisco 24-port SGE managed switches make up the "backbone" of the network.
- The Star2Star service device (StarBox) is actually outside of our firewall on its own static IP address, but operates in bridge mode so the firewall is still publicly accessible.
- Externally accessible internal resources (web/FTP server) appear unaffected.
If I can provide any additional information that might be helpful, please let me know.
UPDATE
Since this is Windows 10, I'm unable to simply uninstall the "Client for Microsoft Networks" feature for the adapter (thanks, Microsoft), but I did go ahead and disable (reboot) then re-enable (reboot) it to see if that might have anything to do with anything... Still no luck.
Again, if I remove everything from between the workstation and the wall jack, it works. Anything in between - other than the old phone - seems to be causing the problem. At this point, I'm really beginning to wonder if there's an underlying problem that's always been present in our network that the installation of this new phone system has simply exposed.
The fact that everything worked normally on this (second) workstation for the first day and the problem didn't present itself until the next day is what's really throwing me. That and the fact that replacing the NIC on the first PC did resolve that workstation's issues but didn't resolve the second PC's problem.
YET MORE TROUBLESHOOTING
Looking for network issues, I checked which DC this workstation was using for authentication. I then logged in to that DC and found that it was critically low on hard disk space (less than 1GB free). Since that DC is a VM and thinking that the lack of available space could cause "unexpected" behavior, I expanded the drive to give it an extra 25GB of space. This did not resolve the issue.
Then I found one thing I hadn't seen/noticed before: When I put the phone in-line and checked the network properties, the connection showed "ad.domain.com (2) - (Unauthenticated)". I tried running the PowerShell commands to reset the secure channel for the workstation with the following results:
> Reset-ComputerMachinePassword -Server 'SERVER.ad.domain.com' -Credential [email protected]
Reset-ComputerMachinePassword : Cannot reset the secure channel password for the computer account in the domain.
Operation failed with the following exception: The server is not operational.
.
At line:1 char:1
+ Reset-ComputerMachinePassword -Server 'SERVER.ad.domain.com' -Credent ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (WORKSTATION:String) [Reset-ComputerMachinePassword], InvalidOperationException
+ FullyQualifiedErrorId : FailToResetPasswordOnDomain,Microsoft.PowerShell.Commands.ResetComputerMachinePasswordCommand
> Test-ComputerSecureChanel -Server 'SERVER.ad.domain.com' -repair
Test-ComputerSecureChanel : Cannot reset the secure channel password for the computer account in the domain.
Operation failed with the following exception: The server is not operational.
.
At line:1 char:1
+ Test-ComputerSecureChanel -Server 'SERVER.ad.domain.com' -repair
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (WORKSTATION:String) [Test-ComputerSecureChanel], InvalidOperationException
+ FullyQualifiedErrorId : FailToResetPasswordOnDomain,Microsoft.PowerShell.Commands.TestComputerSecureChanelCommand
Next, I disjoined the computer from the domain, removed its computer account from AD (verified that it was removed from all DC's), then rejoined the workstation to the domain. That has not resolved the issue.
I verified that the DNS servers in use by the workstation are pointing to the DC's, but I manually set all of the IP address information on the NIC just to test things out. It still didn't work.
On a whim, I ran a test by sharing a folder from my own workstation and was able to access it from the "problem" machine without issue by browsing through File Explorer (\\MYWORKSTATION\TestShare).
Next, just to see what would happen, I tried statically assigning the computer's IP address to something completely different than its usual DHCP-reserved address. All of a sudden, everything started working exactly as it should.
The reason we have it on a DHCP reservation is for content-filtering rules, so I went into the DHCP server and changed the workstation's reservation to the "new" IP address (the one I had statically assigned), reset the workstation to DHCP, then rebooted. Everything is still working!
It appears the issue could be some sort of IP address "conflict", but even that doesn't make total sense. I mean, it makes some sense in that the first computer to exhibit this behavior was "fixed" when I replaced the NIC, probably because that workstation doesn't have a DHCP reservation and the new NIC/MAC address forced the computer to pick up a new address. But, there isn't any other device on the network with either IP address that would have been conflicting. Add to this the fact that none of the other seven computers on the network have exhibited this behavior. I'd really like to know what the actual cause of the issue is/was, but if I can try just changing a "problem" user's IP address to solve the problem in the future, I'll go with that.
192.X.X.1-149) to avoid conflicts with the equipment on statically assigned IP's (printers, switches, etc.). The reservations for certain workstations are in place specifically to apply certain content-filtering rules (e.g., allow certain social media sites to certain users who need access to them for business purposes) and are within the assignable range. The phones are not getting any of these IP's - they are all receiving their IP's in the10.X.X.XVLAN's subnet.PING -aeach computer and possibly remote in to check things more carefully)