Debugging CSRSS.EXE Application Error/Crash

Question

For a while now, I have been getting an intermittent crash of SVCHOST.EXE on boot in Windows XP. Because it is intermittent, I cannot reliably reproduce it, nor can I narrow down when it started, and because it is SVCHOST, I cannot determine which service is the one crashing because the dialog is created by CSRSS.EXE, not any of the instances of SVCHOST.EXE.

One thing I know for sure is that if I dismiss the below dialog that pops up when it happens, then the whole system hangs completely, but if I leave the dialog alone, I can continue to work and use the OS as normal, however upon shutting down (at which time the dialog is inevitably closed), the system hangs, resulting in a needed reset.

I used Anonymous Coward’s suggestion of setting each service to use its own instance of SVCHOST instead of a shared one, and found to my surprise that the crashing process was not one of the SVCHOST instances. It was CSRSS.EXE.

Now I’m left trying to figure out why CSRSS crashes sometimes (the vast majority of searches that include filenames just return those worthless basic analysis pages), but I’m not sure how to debug such an integral part of the system short of a messy kernel debugger. (I’ll point out that the system does not BSOD, so there is no dump generated.)

One thing of not is that the addresses seem to always be the same. It has happened about a dozen or so times and the offending instruction is always at 0x76dea383 that tries to read memory at 0x00000000. Obviously the target memory address is useless, but I’m sure there must be a way to use the source address somehow.

What I have tried when this happens is to close every process and stop every service possible. Unfortunately, that does not help narrow down the cause because the dialog remains displayed because it is created by csrss which cannot be stopped without completely taking down the whole OS.

---

Old information:

I have tried figuring out which service is responsible without success. For example, the last time that it happened, I stopped every single service that was possible (and a few that are not normally possible), quit every single app, killed Explorer, and basically stripped the system down to the bare minimum (there were maybe 10 processes left, using ~111MB of RAM and 0% CPU), but it still hung once the dialog was closed. This makes me suspect that it is one of the critical system services, though if it were, then why would the system keep running in the meantime?

Does anyone of a way I can debug this? For example, can anyone think of a way to determine the instance of SVCHOST associated with the error? I checked the Event Log and Process Explorer but could find nothing to help pinpoint it. Is there something I can do with the memory address?

(Yes, I’ve Googled this and found numerous mentions of the Windows Update service, but it is not that because WUAUSVC and BITS are disabled by default. And no, I cannot do the disable-all-services-and-enable-one-by-one method of testing because if it is one of the critical system services, then Windows won’t run at all.)

Answer 1

That looks like a problem that's hard to solve. You are basically listing everything I would have suggested (stopping services, Event Log, Process Explorer...). The only additional solutions that come to my mind would be heavy ones like re-installing Windows.

However, there is this answer to a similar question that links to a Microsoft KB about debugging SVCHOST crashes using "Debugging tools for Windows", perhaps this might be useful.

Answer 2

Before you reply to the error message, find out the process ID (PID) in task manager, then, at a command prompt, run tasklist /svc to find out which services are running under that process id (PID). You may have twenty services running under one svchost instance, but you may get lucky and only have one or two.

Note: you may have to figure out the correct process ID by elimination i.e. run the task list command, answer the error message and then run task list again to see which one has dissapeared.

Answer 3

""""" http://blogs.msdn.com/b/asiatech/archive/2011/08/29/a-useful-tool-to-generate-dump-file-for-a-particular-service.aspx

There are several ways to dump a particular service: 1. Use Debug Diagnostic Tool (DebugDiag) 1.2. 2. Use the following batch file:

FOR /F "tokens=2 delims=," %%A IN ('tasklist /svc /FI "services eq winmgmt" /NH /FO csv') DO SET PID=%%~A adplus.exe -hang -p %PID% -o c:\dumps

""""" http://blogs.technet.com/b/askperf/archive/2008/01/11/getting-started-with-svchost-exe-troubleshooting.aspx

To isolate these services, there are two different approaches with very subtle differences. The first method is to create an isolated process that runs within the same SVCHOST group as it did before, just not in a shared process. The second method is to create a completely separate SVCHOST group. Since we've been using Windows Updates as our example, let's continue on with that.

Method 1: Creating an Isolated Process

What this method really does is modify one of the registry parameters for the service in question from a shared process to an isolated process. The command syntax is fairly straightforward and uses the sc config command set:

sc config type= own.

So for Windows Updates (wuauserv), the command would be:

sc config wuauserv type= own.

Note that there is a space between the '=' and 'own' - you must insert that space. Behind the scenes, what happens is that the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wuauserv\Type registry value is changed from 0x20 (which denotes a shared process) to 0x10 (indicating it has its own process). You can read more about these particular values on the MSDN Article about SERVICE_STATUS_PROCESS Structure. In order to complete the change, you need to stop and restart the service. To change this service back to being a shared service, run the following command: sc config wuauserv type= share. For this change to complete, the machine itself needs to be rebooted. No other parameters are being modified with respect to this service, however when you change the type to isolated, restart the service and then run the tasklist command to get the list of services, you will notice that there is an SVCHOST.EXE process that only contains the Windows Update service.

Method 2: Creating an isolated Service Group

This method is a bit more involved, and involves directly editing the registry. Please remember to back up the registry before making any changes! The process is below:

1.Create a new REG_MULTI_SZ value named WindowsUpdates in the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost key.

2.Add the name of the service (in this case wuauserv) to the value. You also need to remove the wuauserv from the list in the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Svchost\Netsvcs value to prevent conflicts.

3.Now navigate to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet \Services\wuauserv key and change the ImagePath value from %systemroot%\system32\svchost.exe -k netsvcs to: %systemroot%\system32\svchost.exe -k WindowsUpdates

4.Restart the Automatic Updates service and you should now see a new instance of SVCHOST.EXE that only contains the Automatic Updates service.

5.This method can be repeated to isolate multiple services into their own groups.

6.To revert back to the original configurations, reverse the steps above and restart the machine. Use the backup of the registry to ensure that you get the right services back into the proper groups.

An additional refinement to this method would be to create copies of SVCHOST.EXE that are appropriately named for the isolated service - for example copy %systemroot%\system32\svchost.exe to a new file named %systemroot%\system32\svchost_wuauserv.exe. Remember that you will need to make the appropriate modifications to the ImagePath value in the registry that reflect the name of the executable file. By customizing the executable, you can use tools such as the Debug Diagnostic Toolkit that we covered in an earlier post to monitor specific services for crashing. You can also quickly tell which services are misbehaving in Task Manager as well as getting the name of the failing executable logged in the Event Viewer in the event of that service crashing.

""""

Answer 4

Troublshooting Steps

Well I’ve narrowed it down as much as is possible (again, without resorting to kernel debugging):

1. I examined the Services snap-in (services.msc) right after a clean boot to compile a list of all services that run during a normal boot (I already have it down to the minimum, so that certainly helps)
2. As Anonymous Coward suggested, I configured each and every service that runs to run in its own process
3. Next time it happened, I ran Process Explorer and viewed the loaded images (Properties->Threads) in CSRSS.exe
   1. Made a list of threads and their image files
   2. Dumped the stack of each
4. The next, next time it happened, I observed the moment it happens and what was running at that point as well as changes on the screen

Observations

This was the list of images running in the CSRSS process from step 3.1:

winsrv.dll
csrsrv.dll
ati2mtag.sys

In step 4, I noticed that just as the error dialog popped up, some tray icons disappeared, specifically, the icon from ATI Tray Tools indicating the temperature of my video card.

Results

And there it is!

The moment that the error dialog pops up, ATI Tray Tools crashes and goes away. (How did I not notice that before? Especially since I often use the hotkeys to adjust the brightness of the screen.) This goes hand in hand with the fact that ATI2MTAG appears in the list of threads in the offending process as well as a regular and repeating System Log entry stating that \SystemRoot\System32\ativvaxx.dll failed to load.

Summary/Solution

If anyone else has a similar issue and finds their way here, the problem is with the ATI video driver. If you cannot upgrade the hardware, then perhaps updating to the latest service pack for Windows can fix it. (Now, I just need to set all the appropriate services back to shared process mode—where’s that backup .REG file…)