# Problem:

About a week ago, the NTVDM on an XP installation broke.

# Symptoms:

The first symptom was that a 16-bit executable that is run during an auto-run batch file would hang and peg the CPU at 100% until ntvdm.exe is killed, at which point the batch file continues running (albeit without the work that the executable was supposed to have done). The next symptom was that a different 16-bit executable that is run as part of a shutdown batch script would hang, causing the script to fail.

Interestingly, they do sometimes work (~30%). In other words, if a program is run (from the Run dialog) 10 times, it might run correctly about 2-4 times and hang the rest of the time. This is mostly just with simpler programs that print something out or such as opposed to more complicated ones with UIs like edit.com or command.com which seem to always hang. A few select programs that are exceedingly simple (e.g., crlf.com: 578 bytes, ret.com: 1 byte, etc.) always seem to run and return without problem.

# Tests:

Some tests show that the problem is with the NTVDM itself and not just the specific executables (which worked just fine until recently anyway). For example, running edit.com opens a console and displays the editor, but is unresponsive to the keyboard and must be killed. In fact, even running command.com won’t work; it opens a console and shows the prompt then just sits there, unresponsive.

I checked all the related NTVDM files (and numerous other OS files) and they are all correct. I also checked several related registry entires and they all check out as well. The programs run fine on other XP installations (which is not a surprise since command.com and edit.com work as well); so it is that specific machine. The environment has not changed since the NTVDM stopped working, but I checked it anyway and there is nothing overtly problematic; temp and tmp point to existing directories with no LFN names, no broken variables with nulls or such, and while the path is long, it has been such for some time.

Files verified: ntd*, ntv*, dos*, redir.exe, Mscdexnt.exe, *.sys

Registry entires verified: *\Software\Microsoft\Command processor, *ControlSet*\Control\WoW, *\Software\Microsoft\Windows NT\CurrentVersion\WOW*

The NTVDM and 16-bit apps work just fine in safe-mode on the machine in question; it is only in normal mode that they hang.

# Question:

Can anybody think of anything that could break the NTVDM in a manner that produces these symptoms?

(To be clear; it is NOT the common error “The NTVDM CPU has encountered an illegal instruction.” Also, please don’t suggest using Dosbox or other VM software because that is not applicable, and moreover, the programs in question worked before, not to mention that Windows’ own 16-bit programs won’t work.)

-
What if you procmon running command.com on the good system and bad system and compare? –  ckhan May 2 '12 at 6:40
Actually, I had already tried that with the bad system itself to see if there were any file or registry references to non-existing or corrupt items. I found nothing of particular use that way. A comparison sounds good though. ↑ –  Synetech May 6 '12 at 16:31
Could you disable all drivers apart from the minimal set loaded in Safe Mode, and then reenable them all one by one? (Faster way would be to binary search it - enable half, test, if it works enable half the remainder, if not then disable half of those you just enabled.) –  Bob Dec 17 '13 at 17:37
The Safe Mode list is located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SafeBoot\Minimal, and you can disable drivers by going to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services and changing the Start values to 0. This will probably be quite a tedious process, and you might want to consider scripting it if you are able to. –  Bob Dec 17 '13 at 17:43
@Bob, well I don’t think any drivers or services had changed between the time it was working and when it stopped, but since it does work in safe-mode, it is certainly worth doing some tests (in fact, any old, standard entries can probably be skipped). I’ll give it a shot this weekend when I’m on the XP system. –  Synetech Dec 17 '13 at 19:53