Speaking specifically about 32-bit Windows variants, they have had support for more than 4GB of RAM since Windows 2003 variants (and you can also get a kernel hack for Windows 7 to allow you to use all of your RAM in 32-bit). However, this comes at a cost, as you outlined in the first part of your question.
In a 32-bit operating system, the size of a pointer (memory address) is the same as the word length of the CPU, 32-bits, which allows (as you mentioned) a 2^32 = 4GB memory space. Windows also takes a "virtual memory" approach for applications, so each application has it's own memory space.
Since each pointer is only 32-bits wide, each application's pointers can only address up to 4GB of memory, even though the system can support more then 4GB of RAM. As far as I know, this is the only caveat to using more than 4GB of RAM in a 32-bit operating system. In total, you can have many applications using more than 4GB of RAM combined, but any one particular process can only allocate/access up to 4GB.
Back to your question, let's say you have a program that uses 2GB of RAM. If you have 10 instances of this program, that's 20GB. All 8GB of your RAM will be used up, as well as another 12GB of the pagefile. So yes, under 32-bit operating systems, it is more than possible to use up this memory.
if this 32-bit OS machine has 2GB RAM and 2GB page file, increasing
the page file size won't help the performance. is this true?
Increasing the pagefile size will usually not increase performance (unless your RAM and pagefile are set to the absolute minimum, or set so low your computer constantly thrashes). It will, however, prevent your computer from running out of (virtual) memory. Whenever anything needs to be purged to the pagefile, you're already taking a huge performance hit (since the hard drive is orders of magnitude slower then your RAM).