No, Imagine RAM is like your desk, and swap space/ virtual memory is like a filing cabinet. You can put lots of information into a filing cabinet, but it's difficult to access. Any papers you want to use, you have to pull out and put them on the desk; the problem is the desk can only hold so many papers at once.
32-bit machines can only have a desk that's 4GB (2^32, or about 4 billion bytes) big, but they can still have nearly unlimited filing cabinets. They just have to "swap" out papers when the desk is full and they need to look at another page that's in a filing cabinet.
The reason it doesn't apply to swap is because accessing the contents of RAM or executing code from it is limited to the size of the numbers that it can work with. Think of it like you can only have 4 billion words on the desk, because that's as high as you can count, and you need to work directly with words. "Word 3,547 is mispelled. Change word 3,452,780 to 'Hello'. Delete word 6."
Now, each page might have 4,000 words on it. The catch is when you're referencing swap space, you don't reference it by word, only by page. So you can still only count to 4 billion, but you can count to 4 billion pages. 4 billion pages actually holds 16 trillion words - much, much higher than you can count.
64-bit machines can just count higher, so they can put all the pages on the desk at once, if they have a large enough desk. Even if you put a super-large desk in a 32-bit machine, you can still only count to 4 billion, so you have no way to reference all the extra words on the desk.