There can be a lot of factors involved, including link speed, duplex, minimum packet size, round trip time, the latency of the target host, whether or not an intrusion detection/prevention system is involved, and how many round trips the particular scan type requires, how busy the link is, etc.
So let's assume the scanner machine and the target are on the same gigabit Ethernet, and we'll estimate the theoretical minimum time a scan of 65536 ports can take. GigE is always full-duplex, has a minimum frame size of 64 bytes, and after calculating for mandatory inter-frame gaps, has a max frame rate of 1,488,000 minimum-sized frames per second. TCP Syns, Syn-Acks, (empty) Acks, Fins, and Rsts are all minimum-sized frames. On the GigE I'm on right now, the RTT is 0.3ms (300µsec), so let's assume that the target host replies to Syns with either Syn-Ack or Rst within that same 0.3ms latency. Let's further assume that it'll reply with one or the other; that there won't be any "filtered" ports for which we'll just have to wait for some arbitrary timeout.
So the time it takes to send TCP Syns to all the ports is:
65,536 / 1,488,000 = 44ms
Since GigE is full duplex, the first responses would starting coming in as the later Syns were going out, but we'd have to wait 1 RTT between the last Syn sent and the last reply, so add 1 RTT which is 0.3ms, for a total of 44.3ms.
Even if you had a scan that had to do two round-trips, all your network traffic could still be done in less than a second.
Note that if the target host doesn't reply to Syns for some ports, you have to make your own choice for how long to set your "no response" timeout for, in your own algorithm. But you can think of that like the RTT. Let's say you want to use a 10-second timeout. You can still send all your Syns in 44ms, but you may have to wait 10 seconds from the last Syn sent in order to determine that you're not likely to get a response from that port. So even in the case of all 2^16 ports nonresponsive, and a 10-second timeout, your total time would be 44ms + 10 seconds = 10.044 seconds.