7

I have a server and a client PC directly connected to the same desktop switch. But I also want to put a dedicated separate high speed link between them as well. The aim being to connect them to the internet and LAN with 1G but traffic between PC and server is at 10G:

enter image description here

If relevant, the PC runs Windows 10 on an Asus workstation board, and the server runs FreeBSD 12 on a Supermicro board. Both have Intel 1G and Chelsio 10G NICs, and both baseboards have fully featured workstation/server firmwares. Both are in the same LAN-wide subnet at present.

My questions:

  1. What are the more correct or easily implemented way(s?) to configure the PC and Server's networking, so that the loop doesn't cause networking issues? Meaning, traffic between the PC and Server can only go via the 10G link, and traffic to/from any other devices can never go via the 10G link?

    (So that if I disconnect the wire between PC and switch, the PC isn't still connected to other LAN devices via the loop through the server)
     
  2. If in future I upgrade the switch to a fully managed switch that can handle access controls, such as Netgear ProSAFE, or I start to use VLANs on it, does the "correct answer" change?

Update

As the comments are getting long, this is a summary of some of the issues that they have revealed, which have made this a difficult question.

The main issue is that the network is DNS based. That's important because IP addresses don't show up in Windows file explorer, only discovered host names do. An IP address can be used for a SMB file server and its shares, but it will have to be manually entered every time a file share is accessed, and many programs unfortunately don't have scope to enter a manual path, its expected that all browsable devices will appear in a populated file explorer navigation tree, or descend from one of a list of discovered devices. So using the 10G IP for the server isn't a reliably viable solution.

(Discovery is presumably DNS or WS-discovery from the PAC, and DNS/WS-Discovery/mDNS-bonhour from some other devices. SMBv1/NetBIOS/WINS are all disabled - thanks for noting this aspect @user1686)

The fact that the server has 2 unbridged NICs, one of which has no link to the DNS server and is direct point to point, presumably means its IPs need to be distinct. (And the 10G IPs will need to be static, as there's no DHCP on that link/subnet).

I also suspect I have to block discovery via the 1G link separately if it doesn't use DNS, or configure discovery to use DNS and no other method - from memory one can configure Windows with which discovery methods are enabled and what order to try them in, and specify DNS only? Again thanks @user1686

From the answers, I think I see perhaps 3 possible solutions. Which would be more correct, or are they all valid?

Possible solution 1?

What i get from the answers so far is that if the 10G are in a different subnet and the PC hosts file is manually hacked to override DNS and use the 10G IP for the server and routing or firewalling is set up that ensures the servers 1G IP isn't reachable from the PC, that sounds like it would work.

But its incredibly clumsy and feels like a "hack job". Too much manual overrides set on individual machines. I don't know if that's "normal".

Possible solution 2?

Another option relies on the local DNS running on Unbound, which (like BIND) supports views. That means the DNS server can be configured to give the usual IP of the server to all devices at lookup, but the servers 10G IP to the PC (only). I'm not sure if that's sufficient - is it?

If it is then a solution might be: LAN subnet 10.0.0.0/8, server 1G IP allocated by DNS to say 10.0.0.1 server 10G static 10.0.0.2, and then use DNS to provide the server's 1G IP to all devices except the PC, but return the server's 10G IP + the PC's 10G NIC as its first hop/gateway, to the PC.

Naively the result would be: The 1G and 10G IPs are on the same subnet so the server is seen as being on the correct subnet by the PC. But the 10G IP is invisible to all other devices, and the 1G IP is visible but ignored by the PC, so no conflict arises. So its almost all done in DNS.

Is this workable?

Possible solution 3

Same IPs as above (2), but configure DNS to return both IPs to all devices. Non-PC LAN devices will fail to find 10.0.0.2 and eventually decide that 10.0.0.1 is the IP used. The PC gets firewalled so it drops all packets to 10.0.0.1, and thus decides that 10.0.0.2 is the only working IP (it may need a first hop hint). So again its all done in DNS barring a firewall entry to block the 1G loop.

Workable as well?

Improve this question
5
  • What is the reason for the complicated hybrid with two different connections for a computer? Is it a high price of a switch with 10 Gbps ports?
    – pabouk - Ukraine stay strong
    Jul 8, 2020 at 10:42
  • Yes. Or rather, I have a switch with a couple of 10G, but switches with more than 3 or 4 10G links are prohibitively expensive, so I can't just make all the machines needing fast data, 10G links, due to cost. But Chelsio T420/T540 (2/4 port 10G) are $50-$70 each, dirt cheap compared to an 8-12 port 10G switch. So this question is to understand how to add a point to point 10G link to an otherwise "flat" 1G LAN using one port, and not mess up networking functionality. I can repeat for other devices. Impractical for a datacentre but feasible for home with 4-5 PCs/workstations and one server
    – Stilez
    Jul 8, 2020 at 11:04
  • Why not list only the 1Gbps addresses per DNS and add static routes to the two hosts to the 1Gbps IP address of the ther host via the 10Gbps address of the other host?
    – Hagen von Eitzen
    Jul 8, 2020 at 13:05
  • Because as I understand it (could easaily be wrong), that implies one of 2 failing scenarios: either you can reach the servers 1G via its 10G which implies they are bridged, and you probably wouldnt want to bridge them according to answers, or you cant reach the servers 1G via its 10G in which case your packet will get to the 10G port (because you've set that as a static route, or at least it'll ask the 10G if it can route it towards the 1G NIC) but then hit "unreachable destination" for the 1G NIC IP as packets at the 10G NIC can't reach it. If thats wrong I would want to know?
    – Stilez
    Jul 8, 2020 at 14:11
  • I can't speak to Windows, but I'm sure it can be done -- set up a new subnet between the two machines, it doesn't matter what, but do not set a default route on that subnet. We'll call it subnet 10G. Now set a /32 route to the other side's 1G address via the 10G remote IP. on both machines. They should route packets to the 1G address via the 10G link now.
    – mikebabcock
    Oct 30, 2022 at 15:27

3 Answers 3

Reset to default
13

What is the correct way to configure the PC and Server, so that the loop doesn't cause networking issues? Meaning, traffic to/from other devices can only go via the switch as normal and neither device is seen as a bridge or route to the other?

Just connect the two using a cable and configure both ports to have IP addresses. This link should use a completely separate subnet from your main LAN, e.g. if the 1G switch carries 192.168.1.0/24 then the dedicated 10G link should be 192.168.2.0/24 (or even better, a /30).

Bridging will not occur unless you deliberately set up a bridge. Neither Windows nor FreeBSD (nor any other OS) enable it automatically.

Routing will not occur unless you deliberately set up packet forwarding. Neither Windows nor FreeBSD enable it automatically.

(Additionally, IPv4 uses explicitly configured routing tables, so even if the PC or the server did have routing enabled, routing still wouldn't occur because other hosts would have no way of knowing that they can use them as gateways. IPv6 would need a little more care to ensure that the machines do not send out Router Advertisements, but that's also disabled-by-default on both Windows and FreeBSD.)

Of course, if you don't want the PC to use the server as a gateway, then do not fill in the "Default gateway" field when configuring IP addresses on your 10Gbit NIC...

If in future I upgrade the switch to a fully managed switch that can handle access controls, such as Netgear ProSAFE, or I start to use VLANs on it, does the "correct answer" change?

No.

The main issue is that the network is DNS based. That's important because IP.addresses don't show up in Windows file explorer, only discovered host names do

This discovery isn't DNS based. It might be using LLMNR & WS-Discovery, or the older NetBIOS (if SMBv1 is allowed), or mDNS & DNS-SD (but Windows doesn't actually support that yet), or even LDAP (Active Directory), but it won't be using ordinary DNS.

An IP address can be used for a SMB file server and its shares, but it will have to be manually entered every time a file share is accessed, and many programs undortunately don't have scope to enter a manual path, its expected that all browseable devices will appear in a populated file explorer navigation tree, or descend from one of a list of discovered devices.

If you're otherwise okay with manually entering paths, then you can map those paths to a drive letter using "Map Network Drive" or net use. They will permanently show up under "This PC" subtree.

(And the 10G IPs will need to be static, as there's no DNS on that link/subnet).

DNS does not handle address allocation. I think you mean "DHCP"?

Yes, normally such point to point links don't have DHCP, but on the other hand, there is nothing preventing you from running a DHCP daemon on the server and allowing at least the PC to acquire its address dynamically. (But do it with care – do not advertise the 'Routers' DHCP option.)

or else the server's 10G NIC must bridge to the 1G NIC (gives one unique IP for the server, but breakes everything else!)

Bridging is not really a good solution even with STP to avoid loops. There is no way to tell STP to use a specific path only for accessing certain MAC addresses – either the link is active or it's not.

(There's a reason why they don't call switches "Ethernet routers".)

I guess the newer TRILL or 802.1aq "shortest path bridging" protocols would indeed work here (as they do route at MAC level), but it looks like it will still be years until they begin showing up in affordable consumer switches. (The PC and the server would need to participate too, and for Windows that's even more unlikely.) Until then it's STP all-or-nothing.

If it is then a solution might be: LAN subnet 10.0.0.0/8, server 1G IP allocated by DNS to say 10.0.0.1 server 10G static 10.0.0.2

The two links really need to be on different subnets, otherwise you'll need more things to override on the PC...

The problem with this approach is that when a host has multiple interfaces belonging to the same subnet, it will usually not attempt to guess which one to use for each individual destination IP address. It will just always prefer one interface for the whole subnet. Unless you manually add more-specific routes to override it per destination.

(Windows might actually do guess, though – I haven't checked recently. But I do believe it only applies this guessing to designated "link-local" address ranges.)

the PC hosts file is manually hacked to override DNS and use the 10G IP for the server and routing is set up that ensures the servers 1G IP isn't reachable from the PC

Aron's suggested route does not make the 1G IP address unreachable. Rather, it makes the 1G IP address reachable through the 10G link.

So really you don't need both – you only need one or the other.

Too much manual overrides set on individual machines. I don't know if that's "normal".

If you go with Aron's suggestion, there is only a single thing you need to override: the routing for the 1G addresses of both computers.

(It's unfortunately "normal" for hosts to not participate in IP routing protocols – Windows does not speak OSPF for allowing route costs to be determined automatically, and neither does your LAN router I'd suspect. Windows does speak RIPv2 but I'm not sure if that would be useful in this situation, or indeed if it wouldn't be more work than a static route override.)

Same IPs as above, but configure DNS to return both IPs to all devices. Non-PC LAN devices will fail to find 10.0.0.2 and eventually decide that 10.0.0.1 is the IP used. The PC gets firewalled so it drops all packets to 10.0.0.1, and thus decides that 10.0.0.2 is the only working IP (it may need a first hop hint). Workable as well?

This may work – again, the addresses need to be on different subnets though.

And if you're fine with manual firewall rules, then you should be fine with manual routes as well – they're the same amount of "overrides" and are a slightly better solution. (And aren't they literally the "first hop hint" that you speak of?)

Improve this answer
18
  • 4
    No, it's not a problem at all. Most protocols do not care in the slightest (SMBv3 does if you have multichannel support active, but it was designed to use this to its advantage). But the IP addresses really should be a completely separate range from your main LAN subnet, precisely because the two networks are not bridged.
    – u1686_grawity
    Jul 7, 2020 at 14:47
  • 3
    @Stilez: For L2, Spanning Tree Protocol is only necessary if both ports of a device are bridged. (e.g. if it's a switch, or if you set up a FreeBSD bridge(4) interface). But if frames aren't actually forwarded from one interface to another, then a layer-2 loop cannot possibly occur.
    – u1686_grawity
    Jul 7, 2020 at 16:01
  • 2
    @Stilez: For L3, routing is always based on explicit routes, there is no automatic 'learning' or 'flooding' in IP like there is in Ethernet switches, so IP loops won't occur automatically even if packets are set up to be forwarded between two interfaces (which, again, by default they aren't). IP routing protocols help with avoiding mistakes, but they don't prevent some kind of 'default' behavior -- they work by adding routes, unlike STP which works by blocking paths.
    – u1686_grawity
    Jul 7, 2020 at 16:09
  • 1
    You can also try to explicitly set a route to override the default route, so that the two machines will use the new link to talk to each other. You can also set the hosts files to override the IP for a similar effect.
    – Aron
    Jul 8, 2020 at 2:44
  • 1
    @Stilez The point is that you want to have "next-hop" for "Server" to resolve to 10GBe NIC rather than the GBE Nic. So you could try to set a Route for 192.168.1.xx/32 next hop 192.168.2.xx (notice the /32). This will tell the kernel to put Ethernet Frames onto the 10GBE NIC rather than the 1GBE NIC to try to get to the Server. Terribly hacky...
    – Aron
    Jul 8, 2020 at 4:32
5
  1. Question: No bridging or routing will occur by default. You have to actively enable it.
  2. Question: No.

Easiest configuration: Use a different subnet for the 10 GbE link. /30 will be enough. Some devices even work on a /31 subnet. Use a subnet with addresses that do not have to be accessible through your 1 GbE connection. With different subnets, no packets will randomly go the one or the other way.

As you will probably have DNS on your 1 GbE connection, the easiest way to prevent ambiguities would be to not use names for your 10 GbE subnet. As it is point-to-point anyway, just use IP addresses to configure services that should use the 10 GbE link.

Improve this answer
5
  • 3
    Yup - the only gotcha is DNS / naming. If OP refers to a box by a hostname that resolves to the IP address on the 1Gbit port, it won't use the faster 10G link. So some fudgery needs to be done with a hosts file on the windows box, or only use IP addresses to force the 10G link.
    – Criggie
    Jul 8, 2020 at 4:02
  • Windows is one of the very few "devices" that does not support RFC3021.
    – Jörg W Mittag
    Jul 8, 2020 at 4:02
  • 1
    @Criggie True. Added a recommendation.
    – stackprotector
    Jul 8, 2020 at 5:05
  • 1
    IP.addresses don't show up in Windows file explorer, only discovered host names do, AFAIK. An IP address can be used for a SMB file server and its shares, but it will have to be manually entered every time a file share is accessed, and many programs undortunately don't have scope to enter a manual path, its expected that all browseable devices will appear in a populated file explorer navigation tree, or descend from one of a list of discovered devices. So IP usage isn't optimal - part of why this question is in fact tricky to.discern what's correct.
    – Stilez
    Jul 8, 2020 at 9:27
  • Question updated to include some of the material from comments - thank you! You can evidently see where I am getting confused about the routing issues. If its possible to.shed light this wouldnbe amazing and deeply appreciated
    – Stilez
    Jul 8, 2020 at 9:52
0

10gb lan call it intrAnet. I.e. xyz.local 10.1.1.0/24

1gb call it inteRnet 172.16.1.xcx/24

Set all 10 GB on intranet ((10.1.1.0/24) without any gateway up address

DNS 1. ROUTER_IP (on clients host only)

Server DNS DNS 1= 127.0.0.1 ( on server) DNS2 = 9.9.9.9

On inter lan

Client/server on 2nd interface use IP space for 1gb lan

I.e 172.16.1.0/24 Default gateway and DNS router IP

Router configured with two lans

Intra lan IP 101.1.1/24 No gateway defined DNS1 server IP DNS2 pub DNS (9.9.9.9)

Internet lan ip 172.16.1.1/24 Gateway wan ip DNS1= server ip 172.16.1.xxx DNS2 9.9.9.9

Domain name xyz.local.

For linux/bsd configure /etc/resolve.conf accordingly

All LAN traffic will get resolved with 10.1.1.xxx

Gateway will poke local DNS for xyz.local

All other will go to public dns

Put a

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .