I understand the reason public IP addresses are necessary to be that if we only had MAC to move data between one host and another, the entire Internet would be one massively flat network. But what is the need for private IP addresses? Why can't routers just connect to local clients using their MAC address?
The answer is your software applications and operating system are designed to communicate using TCP/IP, not directly with Ethernet MAC addressing.
So regardless of whether the remote computer/node you wish to communicate with is in your local network or in some other network halfway round the globe, that logical link is established with TCP/IP. That way applications just define the destination IP address in the IP packet, fill it with data (which is actually packed into a TCP/UDP segment), and send it along the way without having to worry how the underlying layers deal with the actual transmission.
It is the operating system's networking stack that determines if the IP address is local or remote:
- If local network, look for MAC address corresponding to that IP address and send the IP packet direct, which is in an Ethernet frame.
- If remote network, look for MAC address of default gateway router to send the IP packet, also within an Ethernet frame. The router retrieves the IP packet from the frame, looks at the address and determines the next node (and its MAC address) to pass it along, and sends it that direction in another Ethernet frame.
Note: it is totally possible for IP packets to be delivered in another data link protocol other than Ethernet (guess what, your dial-up or DSL modem is not connected to your ISP via an Ethernet cable). And applications should not have to care so as long as the destination IP address is reachable. It is also totally possible to wrap other networking protocol data (that is not an IP packet) in an Ethernet frame for transmission.
Ultimately, all packets are delivered to a MAC address. When an IP packet arrives at your router and the NAT table shows that it is destined for your private address 192.168.1.10, the router will do a look up to see what the MAC address of 192.168.1.10 is, and send the packet to that MAC address.
However, a MAC address is a layer 2 address, and by definition only allows communication between items on the local network. So the question of "why private addresses rather than MAC address" assumes that all non-public networks are simple single flat networks.
There are almost no private corporate networks that consists of a single LAN, and many domestic environments have multiple networks. In order to route between these networks, you need a layer 3 protocol such as IP.
So a different way to phrase your question would be "Why don't router manufacturers and OS developers create a layer-2-only local network in the specific cases where layer 3 is not absolutely necessary?". And the answer to this question is that it would not be worth creating a niche solution to a scenario where the existing general solution already works perfectly. There would be no benefits whatsoever.
If MAC were to be used for routing, your routing table would contain routes to each node on network because MAC addresses are different for every manufacturer.
IP is like a standard platform for network devices from different manufacturers to communicate. And with use of IP, you can summaries a route to many devices on a network saving bandwidth and resources. The OSI model provides more insight to how networks work.
Without this standard, I believe networking itself wouldn't be developed as it is today.
Network communication is divided into multiple layers. The figure here shows OSI model. Each layer has a purpose of its own. From the bottom of diagram
i. Physical layer: this layer is the actual medium which connects devices, it may be a cat5 cable,optica fibre etc. .
ii) Data Link layer: this layer takes care of physical addressing and packet switching , checksum etc.
iii) Network Layer: This layer takes care of logical addressing and routing packets.
and so on
This helps separate various functions of the network stack . so that layers are not dependent on other layers.
Now coming to your question.
MAC address is at data link layer, whose function is physical addressing . Network layers function is to provide a logical address to a communication node, So when you are communicating between two devices they have to be addressed by their network address. Also they may not even have the same data-link layer i.e therefore one of the nodes may not even have a MAC address.
For IoT, the use of IP addresses is generally more of a problem than a help. First a segment only has 253 IP addresses, available. This was OK when a typical home had 2 or 3 computers. But now it may have 50 or 60 devices, and by the time you include phones, tablets, laptops, streaming media boxes etc.
When IoT gets traction, and here I mean for real applications like alarms, climate control, etc., not talking fridges, you could easily have several hundred. This would mean segment bridges, and a whole new level of complexity for a home LAN.
Additionally, if IoT devices could communicate by MAC address, and never have an IP address, they are invisible to the outside world, as was pointed out, the "I" in IP stands for "Inter". This means that they don't need a security layer, authentication, ARP/RARP, etc. Greatly simplifying and making IoT devices more affordable. It is likely that some parts of an IoT infra structure, like say the alarm panel of a home security system would need to get an IP address so it could communicate with the outside world. But only that component would be burdened with the IP, security, dhcp, etc.
So, perhaps the answer is to have a "Level 0" and a "Level 1" IoT devices with level 0 supporting only MAC address communication, and Level 1 being a full fledged IP device?
We should also move away from the 48 bit MAC address to the 64 bit MAC address, and perhaps expand it to 12 ~ 18 bytes. The lower 8 bytes would be the 64 bit MAC address, and the upper bytes would be a Dewey decimal system of device function and capabilities. This would allow networks to be self organizing so that expanding say an alarm system, or adding video display, would be transparent (one hesitates to say Plug'nPlay, because of the associated trauma) to the user.
The biggest impediment to this is that switches use MAC addresses but get really excited if the device doesn't have an IP address. But this is an opportunity for network vendors to sell us a whole new generation of switches that support Level 0 devices, :-).
Routers resolve ip to mac at each step because we need mac address no matter what to send packets. But if we start sending data by using mac address of the destination, we would have to search the entire network. Ip addresses logically locate machines in a network. Imagine a scenario where we are in a sub-network 'B' and wish to communicate with a machine in sub-network 'A'. With logical addressing a group of machines in a sub network 'A' and 'B' have ip addresses in sequences. This may not be true with mac addresses, the machines in 'A' or 'B' may have varying mac addresses with no relation between them. Ip thus helps us decide which way to go from host to destination and tells us where the destination is.
Hope this helps.