No, it's not possible for the AP to discover what passphrase a user entered incorrectly.
If you think about it, it were possible, then stealing Wi-Fi passphrases would be the easiest thing in the world. The hacker just has to put up his own impostor AP publishing the same network name, and when people try to join it, he can steal their "bad" passphrase attempts, which are probably the "correct" passphrase for the real network.
The 4 way handshake doesn't exactly involve the passphrase. It is never sent across the air, not even in encrypted form. The handshake is designed such that it wouldn't work out, mathematically, unless both sides already know the correct password.
- The first two messages are where the AP (router) and STA (client) each sends the other a randomly generated number known as a "nonce".
- Both sides are able to use these two nonces, along with the passphrase and network name, to derive the Pairwise Temporal Key (PTK) that they will be using for this session.
- For message 3, the AP encrypts the Group Temporal Key (GTK; the multicast/broadcast encryption key) using the PTK it derived.
If the STA can't decrypt the GTK from message 3, it knows it doesn't have the same PTK as the AP is using. So one of the following must be true (listed in descending order of likelihood):
- The STA got the passphrase wrong.
- The AP is misconfigured.
- The AP is an impostor.
- There's a bug somewhere.
By the way, note that anyone who captures those two nonces with a sniffer, and already knows the network passphrase, can decrypt all the traffic sent and received by that client during that session (assuming they aren't also encrypted at a higher layer like SSL/TLS or VPN). So your traffic on a WPA2-PSK (WPA2 Personal) network is not terribly secure if people you don't trust already know the passphrase, as may be the case on a public/guest Wi-Fi network in a coffee shop, store, hotel, etc.