Your virtual memory is limited by the maximum size of your paging file. This defaults to be managed by the operating system and will grow as is needed, so you will probably only run out when your hard disk is full.
You can check to see whether you are using these settings or whether they have been changed via
Right-Click on my My-Computer and select properties.
Advanced system settings.
Settings (on performance section)
Change (on Virtual Memory section)
When windows gets close to the limit, it will display a warning and recommend that you close some of the programs that are using the most memory.
If you ignore these errors and your system continues to use more memory, then eventually some program will try to allocate memory which will fail. What happens next depends on the program, ie if the program is specifically written to cope with that request being denied, it will fail cleanly, otherwise it may simple crash.
For example, if you try to open another large file in PhotoShop, it may display a message saying that it can't open it because the system does not have enough memory but still allow you to continue to work with the files you are already working on. But if you are totally out of memory, you may not have enough free to display this message and maybe not enough to display the message asking you if want to save your files when you try to close.
So, in my experience, there are just too many things that fail badly when you get to this point, that you are really should heed the warning, but even if you ignore it, you certainly should not expect a BSOD.