Ever since computers went from being 16bit computers to 32bit computers people have started seeing this error message. As stated by SamB - there are some ways in which you can coach the program in to running but ultimately - you will see this message again and again. The real problem is, as SamB talks about, that there are now so many other programs that run in the background that the 16K limitation is exceeded just by bringing up your computer. Under a 32bit OS this problem doesn't appear because you have so many more handles that can be worked with. BUT! Your 16bit application can not handle that. It was written to run under a 16bit OS. That means it was meant to run under either Windows v3.1 or Windows v95. Windows v98(se) or Windows vME both are 32bit OSs so even there you would run into this problem.
My solution is - download either Virtual PC from Microsoft (it is free) or download virtual Box from Oracle (it is free too) and install a copy of Windows v3.1 or Windows v95 and run the program under the OS it was meant to be run under.
Now, unfortunately, if you did not keep your Windows v3.1 or Windows v95 diskettes (or CD in Windows v95's case) then you will have to find and buy them. Or you can petition Microsoft for a copy but I do not believe they either kept them or will release them. (Unlike Apple Computer Inc. which has released, for free, DOS 3.3, ProDos, ProDos for the //gs, and Mac OSs up to version 9.x)
So this is my suggestion/answer. For Windows v3.1 and Windows v95 I would lean more towards Virtual PC as it runs under Windows XP really nicely and it also goes all the way back to just using DOS, DR DOS, and even IBM PC DOS or BDOS.
To better answer SamB's question about incompatibility (which I did not say). Let's take a look at an 8bit OS. The 8bit OS expected all of the various stacks (call stack, handles, whatever) to be a certain size and to not exceed that size. When you exceeded it - you would get a blue screen of death or maybe your system would just reboot. When 16bit OSs came along the 8bit programs all acted still as if they were running on an 8bit OS system. This meant that if they exceeded the 8bit allocation size they probably would still give you a blue screen of death or your system would still just reboot. MANY companies never cared about whether or not they were crashing computers so long as the customers didn't complain about it. (Commonly called the squeaky wheel syndrome.) Microsoft tried to alleviate these problems by testing as many programs as they could for each OS but the thing is - some programs just would not work under a newer OS. My suspicion (and NOT FACT) is that some of these early stacks were jamb up against another stack so if you overflowed one stack it could adversely affect a different stack. Remember that memory was not cheap like it is today so you tried to cram as much into memory as possible. I, myself, managed many times to crash my Apple ][+ as well as my IBM PC XT simply because I caused the stack to overflow. One of the other problems is (as I said) that there were a lot of bad programmers out there who did not care one bit about cleaning up their programs. So they would allocate memory, stack memory, handles, and all sorts of other things and not release them. Your only hint that this was happening is that your computer would start to run slower and slower until you rebooted it. Rebooting would clear everything out of memory, give all of that back to your OS, and you OS would then run faster again. As I said - it is similar to how TSR or Terminate and Stay Resident programs would work. In today's OSs - many times processes are isolated from other processes and the OS itself takes care of releasing memory back to the pool (as well as clearing the stack or even allocating a new stack to each process so that each process can muck around its own stack but not the OS's stack). The problem though - is older programs from like the 16bit OS era were statically oriented. So the libraries et al were included in to the program. This means it might attempt to override what the current OS is saying it can do. So a 16bit program might say "write to location XYZ" but "XYZ" is a part of the newer OS. So then you've overwritten who knows what and your system can hang or crash because of it. If you also have admin privileges - you REALLY can muck things up. Thus my suggestion to use Virtual PC, re-install the earlier OS - and run the program under the OS it was meant to run under. First - you stand a better chance to be able to run the program there and second - if it crashes all you lose is the virtual system and not your entire computer system.
But it isn't "incompatible" as much as it is "archaic". To really, really simplify you can think of it like this: 8bits = 8K stacks, 16bit = 16K stacks, 32bit = 32K stacks, and so forth. It isn't accurate but it does get across that the different OSs used different sizes for everything. Which is why you have problems when you take something from one OS to another.
Does that make sense?