[NAT] routes the requests out to the web and back to the client that requested it
right, but as a point of clarification on how that works: NAT causes your private IP addresses to look like a public IP address, typically before the traffic leaves your network. So, the web doesn't route the traffic back to your private IP address. The web returns the traffic to the public IP address where you are seen.
Issue number 1:
I think if you understand how NAT works in detail, this will help answer some of your questions. The most typical implementation that I've seen is NAPT ("Network Address Port"-based Translation). When you send data from a private address, the router takes that private address and adds that information to a chart/table in the router's memory. The router also chooses a TCP or UDP port number (probably selected rather randomly), and keeps track of that in the same chart/table. The router then sends the information to the Internet using the public IP address, and identifies the "source port" number as the number it chose. (Both TCP and UDP use two port numbers: a "source" number, and a "destination" number.) When the destination (say, a web server listening on TCP port 443) notices the traffic going to TCP port 443 and coming from a different TCP port (e.g., TCP port 53874), that server may reply by sending traffic back to the originating router's public IP address on that same TCP port (e.g., TCP port 53874). The router then looks up information in its table/chart, and knows which private IP address to send the information to.
If you just chose a private IP address at random, the router should not have that private IP address in its table/chart, so that wouldn't work.
I'm assuming there's another means to route data to the correct client happening in the router/NAT that isn't just IP, but I'm not sure how this is done.
No.
I mean, well, yes, there is. There is some other routing that can occur, and so I will just quickly acknowledge that. There's use of 802.1q "VLAN" tagging and other technologies that can be used for some data/signaling clouds, and could impact how traffic ends up travelling through a network. However, those additional routing methods are typically deployed for purposes of speed, security, or compatibility (mainly with non-IP networks, like maybe some older networks inside a phone company). You don't really need to think that these advanced professional-level networking techniques are going to introduce a new and necessary part of understanding how basic routing works, because basic simple IP routing can be used to explain what you're asking about.
Issue #2: The router's internal defenses
Now, let's just pretend like the router isn't performing NAPT, so we're just going to ignore the whole table/chart that matches public TCP/UDP port numbers to private addresses. Let's just pretend like you're sitting at the customer's ISP, and want to maliciously send the packet through the router to the customer's private IP addresses.
Most routers, performing some very basic firewall-like actions, will notice that the incoming traffic is arriving on the port labelled "WAN" (which stands for "Wide-Area Network"). To provide some protection, the routers will realize that incoming traffic on that network port should be using the router's public IP address, and certainly not any of the common "private" IP address ranges.
So, the router will defend the network by dropping the packet.
Issue 3: ISP's protect us better
The standard for any ISP is to not allow any traffic to or from the private IP address ranges (which are the network address ranges identified in IETF RFC 1918 Section 3 for IPv4, or addresses starting with "fd" for IPv6).
So, if you are trying to send traffic from your home, through your ISP, and then through the rest of the main Internet backbone of ISP's, then through the victim's ISP, then through your victim's router, and finally to the "private IP" address of your choice, then you ought to fail. This is because ISPs typically follow the convention of blocking traffic involving the private IP addresses. (ISP's would be crazy not to do this.)
In the above scenario, the protection isn't provided by the target victim's ISP. Granted, the target victim's ISP is likely set up to drop the packets, thereby protecting the target victim's router. However, your ISP is also likely to drop the packets, so the packets won't even reach the target victim's ISP.
Issue 4: ISP Motivation
Why would ISP's would be crazy to allow traffic involving private IPs?
Well, keep in mind that all organizations are allowed to use the private IP addresses for their own internal networks. That includes ISPs. Your ISP might be using private addressing for some of the ISP's local equipment. You shouldn't be able to notice/detect whether they are doing that or not.
The way that ISP's route traffic is that they rely on "routing tables" that determine where the traffic should go to next. These "routing tables" do not specify which other ISP will be happy to receive traffic that has a private address as a destination. The ISP will have nowhere to send the traffic to, except possible to some of its own internal equipment. Since a lot of traffic involving private IP addresses are likely to be malicious or otherwise problematic (perhaps involving traffic coming from a device that is set up incorrectly), the ISP certainly doesn't want such traffic to be sent to its own internal equipment (and possibly causing problems for the ISP). Therefore, the ISP will want to block packets going to a private address immediately, before allowing the traffic any further through the ISP's network.
If an ISP broke these rules, the ISP would likely be inviting malicious traffic that would likely have no effect (other than using bandwidth), but if there were any effect, that would likely be bad for the ISP.
Issue 5: What would really happen
So far, I've explained why the Internet will block the traffic. But let's look at what is really going to happen.
If you're sitting on your computer, and you decide to send a malicious packet to a private IP address, what's going to happen?
If the traffic got as far as your local ISP, that local ISP would block the traffic immediately, as already explained.
However, what's really likely to happen is that the traffic won't get as far as your local ISP. Instead, your local router will figure out what to do with that traffic.
If you are not using the same "private" IP address yourself, then your own router will probably not know what to do with the traffic, so the traffic will be discarded.
Otherwise, if you are using the same "private IP" address yourself, then you will likely be attacking a device on the network you are on. If you are the administrator of that network, then you'd be attacking your own network!
In any case, your router is not likely (when using common default configurations) to pass the traffic on through the Internet.
Review:
All of these issues are about why the Internet won't allow your attack to a private IP address. If you are actually on the same local network as your target, you might not even need to go through a router, so none of these defenses will be in your way. However, if you are located elsewhere on the Internet, these defenses probably will get in the way of an attempt to make such an attack.
Although you can violate certain standard rules with (at least some types of) routers that you may be able to control, other organizations are likely to also block private IP addresses, thereby effectively preventing attacks on remote systems.