What happens when a 32 bit program (running on an 64 bit machine) hits the memory limit?
The same thing that happens on a 32-bit system: the program fails in some way.
As far as the 32-bit program is concerned, the "universe" consists of up to 4GB of data. It does not know about the greater area outside of its own little box (think of the fish analogy that Michio Kaku uses in his book Hyperspace).
If you open too many tabs and drive its memory usage up, it will fill up the universe and when it runs out, it will complain that it does not have enough memory to open a new tab or display a picture or whatever. Well, it will complain if it is well written; if it is poorly written, then it will just crash.
Think about it from a low-level, programming point-of-view. The 32-bit program uses 32-bit pointers to store is data. That means it can point to at most 4GB of data. If it has already used up 4GB, then even if the OS can give it more, what would be the value of the new pointer? The new address would be too far away and the program would not be able to put such a large address in a 32-bit pointer.
As an analogy, think of phone numbers. Let's say in your city, there are not too many people, so your phone numbers are all 5-digits long allowing for up to 10,000 numbers, so all you address books have up to 10,000 spaces for phone numbers. The country however has lots of people, so it uses 7-digit numbers. Even though the country can give you the number of a person far away, you would not be able to store it in your address-book because you only have 5-digits of space to print numbers in, so once you have printed 10,000 numbers, the book is completely full.