The theory of the subnet mask is that it defines what part of the IP address is the network address and what part of the IP address is the host address:
10.100.0.1 - IP address;
255.0.0.0 - Subnet mask;
10 - network address,
100.0.1 - host address.
Hosts within same subnet can talk directly to each other. That means if host A and B are located within the same subnet and A wants to talk to B then A will send it's traffic directly to B. If host A wants to talk to host C which is located in a different subnet then A will have to route this traffic to the gateway which knows (hopefully) how to reach different network. So, it is up to the host to define where to send traffic:
- Directly to the host (second host is within the same subnet)
- To the gateway (second host belongs to a different subnet)
What happens in your case is that your "Authorized" clients have IP addresses
10.100.0.10 - 10.100.0.250 (I assume the subnet mask is
255.0.0.0). The server has IP address
10.100.0.1. To a host from the "Authorized" range this server is located in the same subnet.
10.100.0.10 from the "Authorized" range wants to talk to the server - it first checks if this server is located within the same subnet or not. For the host
10.100.0.10 with subnet mask
255.0.0.0 same subnet would be all hosts within the range
10.0.0.1 - 10.255.255.254. Server's IP address happens to be in this range. For this reason a host from "Authorized" range makes an attempt to reach the server directly and (assuming they are located on the same Layer 2 network) this attempt succeeds.
In this case even though server has different subnet mask - it happens to be located in the bigger subnet (which is also a subnet for the "Authorized" clients). If your server will have different second byte in the IP address (
10.150.0.1 for example) it will be unable to reply to the host from "Authorized" range, because from the server's perspective, the "Authorized" range would look like a different subnet and server would need to send traffic to a router. If there would be no router - then there would be no communication.
If you want to separate your network to the "Guests" and "Authorized" parts then you need to make them to be located in the different subnets that do not overlap.
- "Guests" -
10.10.0.1, subnet mask
- "Authorized" -
10.20.0.1, subnet mask
Server would be located within "Authorized" part of the network having IP address
10.20.0.100, subnet mask
With this setup these subnets will be effectively separated from each other, since parts of IP addresses representing their subnet will differ:
10.10 for Guests
10.20 for Authorized
At this point communication between these subnets will be possible only via router that has interfaces in both subnets.
Also, it is worth mentioning, that while all your computers share same Layer 2 network nothing will prevent a Guests to manually assign themselves IP addresses from the "Authorized" range. This will effectively make them to be part of the Authorized network.