Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've got a Dual core processor, and one of the two is constantly at 100%. Looking in ProcessExplorer shows me that it's Deferred Procedure Calls. Reading around the net seems to give me tons of different answers.

Is it possible to set out a couple of steps to trying to narrow down what the problem might be in my case?

Update 1: FWIW, the problem persists even in Safe Mode.

Update 2: I unplugged everything I could from the back of the PC, and that bought me 40% more free processor. I also downloaded the RATTV3 tool, but for some reason on my machine it's not giving me a driver-by-driver breakdown. There's a good description of both DPCLatencyChecker and RATTV3 here.

Update 3:, LatencyMon (see my answer below) tells me it's nvstor32.sys - which is NVidia's SATA driver - with times of about 5300 µs.

Update 4: The plot thickens, while pondering whether to try booting of a recovery disk (to see if it's really drivers, and not a hardware problem), I noticed that the DVD/CD player wasn't working (i.e. Not even opening the door when I press the button). Given that the machine just got back from having the motherboard replaced, I figured maybe they'd forgotten to plug it in. I opened the box, everything seemed ok, but I unplugged and plugged it back in again. On reboot, everything was fine - no more DPC (highest now 300µs)!

Update 5: On following day, problem back, CD player not working again, even the cursor in the password textbox is blinking in slow motion... Tried unplugging everything I could think of, and on second reboot, worked again (as at Update2). Next time I'm going to try unplugging the CD player completely...

Update 6: Just noticed System event log has nvstor32.sys giving an error saying Parity error detected in \Device\RaidPort0, then a warning about sending a reinitialisation. Now just have to work out which one RaidPort0 is... (note, I've got no RAID set-up, it's just a bog standard Acer). Oh, and my Avast setup apparently got killed when I did a System Rollback (or whatever it's called), because it won't start (RPC error), won't uninstall (setiface error has occurred).

Update 7: Finally got time to reboot with DVD unplugged. No more DPC problems! (lots of page faults though, but that's for later). Next step: work out if it's the cable, or the DVD player.

Update 8: Borrowed a SATA cable, booted with it, no problems. CD/DVD player works, no DPC problems with nvstor32.sys, no processors blocked. Happy end... almost: I've still got problems with Avast, apparent DPC problems with storport.sys at start up (maybe normal for USB?), and lots of hard page faults. But those will be the subject of other questions.

Postscript: I recently started having the same problem, and using the same method, managed to track it down to a USB stick (the one I was using for ReadyBoost) being shot.

share|improve this question
2  
Real good tools and help over here...msfn.org/board/topic/… –  Moab Oct 24 '10 at 20:59

6 Answers 6

up vote 32 down vote accepted

Here is the story of how i found the cause of my high DPC latency.


My system was experiencing clicks and pops during sound playback. i knew this meant that something in kernel mode was hogging the CPU. My first thought was to poke around Process Explorer, and see if anything looked out of place. The only thing that caught my attention was an excessive amount of time spent performing Deferred Procedure Calls (DPCs):

enter image description here

i knew that DPCs are code being run inside a driver; the challenge was to figure out which driver. i turned to the DPC Latency Checker tool, which showed me just how bad the latency was:

enter image description here

DPC Latency Checker suggests going through devices in Device Manager, and disabling non-essential hardware one-by-one (e.g. network card, sound card), hoping to isolate the buggy driver. (If you disable a device, and the DPC latency suddenly drops: you've found your culprit!):

enter image description here

Unfortunately, after disabling everything i possibly could (while still being able to use the computer - don't disable your hard drive, video card, mouse, or USB hub the mouse is plugged into!), the latency was still high. Next i turned to the the Windows Performance Toolkit (part of the Windows SDK), and an excellent blog post by Peter Weiland, "Measuring DPC Time". After installing the Windows Performance Toolkit:

enter image description here

i opened an elevated command prompt and ran:

>xperf -on Latency

Note: The Latency group is a predefined set of events that can be traced from the Kernel Group provider:

>xperf -providers kg
   Base           : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+PROFILE+MEMINFO
   Diag           : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PERF_COUNTER+COMPACT_CSWITCH
   DiagEasy       : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PERF_COUNTER
   Latency        : PROC_THREAD+LOADER+DISK_IO+HARD_FAULTS+DPC+INTERRUPT+CSWITCH+PROFILE
   ...

In this case Latency corresponds to the Kernel Flags:

  • PROC_THREAD Process and Thread create/delete
  • LOADER Kernel and user mode Image Load/Unload events
  • PROFILE CPU Sample profile
  • CSWITCH Context Switch
  • DPC DPC Events
  • INTERRUPT Interrupt events
  • DISK_IO Disk I/O
  • HARD_FAULTS Hard Page Faults

After letting that run for a minute, i stopped the trace, and had it save to a file:

C:\Users\Ian\Desktop>xperf -d thingy1.etl

And then i viewed the results of the trace with the command:

C:\Users\Ian\Desktop\xperf thingy1.etl

This loads the graphical Windows Performance Analyzer. Right clicking on the DPC CPU Usage graph, i selected Summary Table. This shows a breakdown of time spent in DPCs by driver:

enter image description here

Right away i can see one driver (tsvp.sys) taking an average of 2.8ms per DPC execution, which is an order of magnitude slower than any other driver:

enter image description here

Googling tsvp.sys gave me the answer: CommView, which i had recently installed.

The question now is how to disable this driver. Using AutoRuns, i can see that it's installed as a driver service:

enter image description here

Using Device Manager, i can disable the service that hosts this driver. First you have to Show hidden devices, then expand the Non-Plug and Play Drivers node:

enter image description here

Finally i could stop the driver service, and i changed it's startup mode from System (meaning the driver is an essential part of Windows, and Windows cannot boot without it), to Demand (meaning i can start the driver when i want to):

enter image description here

Stopping the driver service immediately fixed my DPC latency:

enter image description here

i may or may not completely uninstall CommView, but for now i've solved the Case of the High DPC Latency.

share|improve this answer
6  
+1, nice work Sherlock! –  Moab Feb 19 '11 at 1:52
2  
+10 if I could! This has got to be the canonical answer, exactly the kind of thing that makes SU rock! –  Benjol Feb 19 '11 at 9:36
    
After spending a lot of time researching everything involved, i see now that LatencyMon (which i only learned of from your answer, Benjol) uses the same "Event Tracing for Windows" to be able to hunt down the offending driver. It's great for a "just tell me what's wrong" answer. But i also also believe that seeing the step-by-step reasoning is more helpful. –  Ian Boyd Feb 22 '11 at 13:40
2  
Note! If you need/want xperf on Windows XP, see here: blogs.msdn.com/b/pigscanfly/archive/2008/02/24/… –  Martin Nov 24 '11 at 19:39
    
You might also refer to these following answers if you have problems installing Windows SDK with VisualStudio installed or without it. –  Chuim May 11 '12 at 6:26

PROGRESS REPORT

The best tool I've found so far is LatencyMon, which basically does everything that the preceding two tools do, without making you think. The download page asks you to register via email - but nothing happened for me when I did that - but you can scroll to the bottom of the page to download anyway.

alt text

share|improve this answer
    
Benjol... it's better to update your question rather than post them as answers. –  KronoS Oct 22 '10 at 23:57
1  
@KronoS: This is sort of an answer as well, since it's not duplicating an existing one. –  Hello71 Oct 23 '10 at 0:08
    
@KronoS, @Hello71. You're both right, I guess - this is a work in process (LatencyMon says it's nvstor32.sys). If you're still around, I'm in chat for a few minutes. –  Benjol Oct 23 '10 at 0:42
1  
I'd still encourage you to update your question and copy paste this in there. That way it's easier to follow your progress! –  Ivo Flipse Oct 23 '10 at 3:38
1  
This tool does not work on Windows XP (it says so on the download page and the setup will also warn you). Only usable with Windows Vista and higher. –  Martin Nov 24 '11 at 18:57

In my case I used LatencyMon (from Benjol's answer) and found that the driver freezing life, the universe, and everything was (also) storport.sys which is a Microsoft driver for "high performance buses". That confirmed my suspicious that the problem was IO related.

I also went ahead and looked at my Windows 7 Event Viewer, folder Windows Logs -> Application, and found several batches of errors from Volume Shadow Copy (VSS) happening every 30 min to 2 hours. They're details were like this:

Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {b5946137-7b9f-4925-af80-51abd60b20d5}. Routine returned E_INVALIDARG. Routine details GetSnapshot({00000000-0000-0000-0000-000000000000},000000000023C850). 

Operation:
   Get Shadow Copy Properties

Context:
   Execution Context: Coordinator

I then began investigating what the heck is VSS and what it is used for. I went over several - pages - about - VSS troubleshooting. Going through all of those I had one big suspect: my backup software CrashPlan.

Following that lead, I quickly found a page relating it with VSS errors. By following the instructions there to disable backup of open files, which uses VSS, the freezes, high Kernel CPU usage, etc were completely extinct. And don't get me wrong: CrashPlan is great! Just this feature wasn't working on my machine.

BTW, this page right here was THE ONE that gave me the initial lead that helped me finding the root cause of my problems. Thanks so much @Benjol and all others that answered previously! I hope my answer will also help others...

share|improve this answer
    
Thank Chuim, that maybe my exact problem too, I have been working on resolving this problem for weeks and I've finally narrowed it down to VSS and storport.sys but was unable to find the root cause (CrashPlan backing up open files) until your post. I'm not sure yet if this'll fix it, but it's the best lead I've had for the high DPC latencies so far! –  Matt Palmerlee Jul 30 '12 at 1:19
    
I've just verified that tweaking the crashplan settings to not backup open files worked! Thanks so much! Now I can play Skyrim without horrid audio pauses and glitches! –  Matt Palmerlee Jul 30 '12 at 4:57
    
@Matt: I'm glad my answer helped you! Enjoy Skyrim! :) –  Chuim Aug 7 '12 at 6:36
    
Just want to add that I was experiencing audio stuttering after a new PC build and found that Crashplan was the culprit as well. Found this answer via computercabal.com/2012/07/debugging-audio-skipping-lagging.html. Thanks to all for doing so much work to track this down! –  chucknelson Jul 21 '13 at 14:43

There's probably a device driver that's keeping your system busy. One way to analyze this is to run DPC latency checker. Then disable one driver at a time and see if the DPC load goes down. (Process explorer also works.)

You can disable device drivers in Computer Management -> Device Manager.

share|improve this answer
    
thanks, I'm going to read up on that link. Excuse my ignorance, but which devices can I safely disable without 'cutting off the branch' (i.e., keyboard, screen, mouse etc.)? –  Benjol Oct 22 '10 at 11:24
1  
Not sure, my primary suspects would be non-Microsoft services. I'd just try, if it goes wrong you can boot in safe mode and re-enable the drivers –  Andomar Oct 22 '10 at 11:28
    
OK, I see that page includes a list of drivers to avoid. Have to hope it's not one of them. –  Benjol Oct 22 '10 at 11:28
    
Before that, I think I'm going to try booting from a recovery disk - if I still get the problem, it's more likely to be a hardware thing? –  Benjol Oct 22 '10 at 11:32
1  
+1 for latency checker. In my experience, the most common culprit here is the driver for a wireless networking card. –  Shinrai Oct 22 '10 at 15:12

I feel I should add my answer here because this issue is difficult to solve and not always down to bad drivers or IRQ conflicts.

I had high RPC latency that were causing pops/crackles in my pro-sumer USB soundcard. The tools described in the accepted answer were not helpful in identifying a particular driver that was causing an issue. The latency was occurring across a number of processes: HAL, USBPORT.SYS and the Windows kernel. Digging deeper into these processes did not reveal an obvious culprit.

It turned out in my case that the issue was lower-level and specific to GigaByte motherboards with certain chipsets and BIOS revisions. The solution was to disable Intel SpeedStep and all other motherboard specific features that adjusted CPU speed and voltage on the fly. Once these options were turned off my RPC latency was immediately fixed.

share|improve this answer

I started seeing this error after resolving an IRQ error with my nVidia 10/100/1000 Ethernet controller that appeared when upgrading my graphics card to the GeForce GTX 550 Ti.

It seems after the upgrade to the new GeForce drivers 295.73 and then resolving the interrupt conflict, I had removed, damaged or uninstalled existing nForce SATA/RAID controller drivers. I don't use RAID, the error still persisted, and locked up Vista Ultimate 64-bit from time to time.

After trying all the troubleshooting suggestions I found on the web, a simple solution presented itself...I upgraded to nForce SATA/RAID controller 15.58, but left other nForce drivers alone.

That fixed it for me, and I now have resolved all my driver conflicts. Hope it helps you, too.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.