4

I'm a web developer who is pretty new to the industry. I was presented with a coding challenge for a job interview where I need to design a messaging system and architect a system for how it deals with messages, malformed messages, different message types, status logging, et cetera...

My question is regarding packet size over TCP.

Incoming messages are at a rate of 10,000 messages/sec at a size of 2KB per message. I've been trying to find the max, max-safe, or max-practical packet size limitation. I've seen in several unverified places (i.e. not in technical documentation) that the max theoretical size is 64KB. Is that correct? In that case my example of sending 2KB messages would easily fit into a single packet and reduce the complexity of this system.

If 64KB is an incorrect number, what would the correct number be? Additionally, I'm not simply trying to understand the max theoretical size but the max practical size. I want to cover edge cases where messages may be slightly larger than the targeted 2KB as well as leave room for the various headers that TCP needs.

1 Answer 1

12

Ethernet has always been influential on TCP/IP packet sizes. Ethernet has a standard MTU of 1500 bytes, which, after typical IPv4 header overhead of 20 bytes and typical modern TCP header overhead of 32 bytes (used to be 20 bytes but we add a 12-bytes timestamp option nowadays), results in a TCP Maximum Segment Size of 1448 bytes.

PPPoE, which is popular among DSL-based ISPs, adds another 8 bytes of overhead, so TCP connections that traverse a PPPoE link end up with a TCP MSS of 1440 bytes. Other technologies may add more overhead.

Most modern TCP/IP stacks perform "Path MTU Discovery" (PMTUD) so they never have to rely on IP fragmentation. Unfortunately some sites block the ICMP messages necessary for PMTUD to work, accidentally creating "PMTUD black holes" where PMTUD doesn't work. To allow people behind PMTUD black holes to still connect to their services, Google sites choose to negotiate a very conservative TCP MSS of 1380 bytes (last time I checked).

So I'd say one could make a pretty good argument that if you want to tailor how your application does writes, to make sure they only fill a single packet most of the time, make your writes no larger than 1448 bytes, and perhaps no larger than 1380 bytes.

IPv4 datagrams can be as large as 64 KibiBytes, but exceedingly few paths across the Internet have a 64KiB MTU, so that number is irrelevant to most packet-size planning. Your theoretical messaging-protocol server is probably connected to an Ethernet-like network, which probably uses standard 1500 byte frames, so your own server's IP stack would have to fragment that 64KiB write into 46 or so separate packets before it could even begin transmitting them on the first hop. Even Ethernet networks that are set up to use nonstandard "jumbo frames" usually max out at 9000 byte MTUs. Off the top of my head, I can't even name a physical / data link (layer 1/2) networking technology that allows for 64KiB MTUs. Maybe IP over Thunderbolt.

5
  • So, you're saying that while TCP can theoretically contain up to 64KB of data no network can transmit that size, correct? Jul 18, 2018 at 19:43
  • @MBaka A single IP datagram can be 64KiB, but the vast majority of networks would require the IP layer to fragment that datagram into lots of smaller packets and transmit it that way, with the IP layer networking software on the destination host reassembling them back into the 64KiB datagram before passing it further up the stack.
    – Spiff
    Jul 18, 2018 at 19:54
  • Got it! But if that's the case, why was the IP datagram even designed like that if networks can't support that without fragmenting it further? Jul 19, 2018 at 18:09
  • @MBaka Decades ago when IPv4 was designed, the designers figured MTUs would grow and left room for that growth. It’s kind of surprising that Ethernet’s 1500 byte MTU has stuck around and stayed influential for so long.
    – Spiff
    Jul 19, 2018 at 18:51
  • That makes sense. Thanks for the response and knowledge! Jul 23, 2018 at 0:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .