We have a serious issue with a C# application being terminated silently at random and infrequent points in time on a Windows 10 32-bit installation. E.g. it might be a month between occurrences. Or sometimes just a day.

Basic system specifications:

Microsoft Windows 10 Enterprise 2016 LTSB
Version 10.0.14393 Build 14393
32-bit

Using https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-and-clearing-flags-for-silent-process-exit we have configured silent process exit monitoring. And we finally have a few samples of this:

The process 'APPLICATIONPATH\APPLICATIONNAME.exe' was terminated by 
the process 'C:\Windows\System32\svchost.exe' with termination code 1067. 
The creation time for the exiting process was 0x01d43bd8689073eb.

Looking at the dumps for this, which was setup for the monitoring we got a process ID for the svchost. This service was still running at the system, and it shows the following list of services:

Services

Which seems to be a list of "netsvcs" for Windows. Opening the dump from the svchost.exe and looking at this a single thread was found with an interesting call stack:

ntdll.dll!_KiFastSystemCallRet@0 ()
ntdll.dll!_NtWaitForSingleObject@12 ()
ntdll.dll!RtlReportSilentProcessExit()
KERNELBASE.dll!TerminateProcess()
ubpm.dll!_UbpmpTerminateProcessCallback@12 ()
ubpm.dll!UbpmUtilsTimerCallback()
ntdll.dll!TppTimerpExecuteCallback()
ntdll.dll!TppWorkerThread()
kernel32.dll!@BaseThreadInitThunk@12 ()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8 ()

UBPM is the Unified Background Process Manager. But how can this be terminating our application? And why? And what does the termination code 1067tell us?

Below is the log entry from Silent Process Monitoring:

Log Name:      Application
Source:        Microsoft-Windows-ProcessExitMonitor
Date:          2018-08-31 15:26:09
Event ID:      3001
Task Category: None
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      PC
Description:
The process 'APPLICATIONPATH\APPLICATIONNAME.exe' was terminated by the process 'C:\Windows\System32\svchost.exe' with termination code 1067. The creation time for the exiting process was 0x01d43ed2aee892ab.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-ProcessExitMonitor" Guid="{FD771D53-8492-4057-8E35-8C02813AF49B}" EventSourceName="Process Exit Monitor" />
    <EventID Qualifiers="16384">3001</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2018-08-31T13:26:09.988216500Z" />
    <EventRecordID>4853</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>PC</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData Name="EVENT_PROCESSTERMINATION_CROSSPROCESS">
    <Data Name="param1">APPLICATIONPATH\APPLICATIONNAME.exe</Data>
    <Data Name="param2">C:\Windows\System32\svchost.exe</Data>
    <Data Name="param3">1067</Data>
    <Data Name="param4">01d43ed2aee892ab</Data>
  </EventData>
</Event>

NOTES: The PC is not being shut down at the moment the app terminates nor are there any other indications in event logs as to why the process was terminated.

UPDATE 1: Here a few extra details (trying to answer as many as in comments):

  • Process is (sometimes) started via TaskScheduler when Windows is started yes. Other times by user. Not entirely sure problem only occurs when started via TaskScheduler. But interesting point? Could windows kill a task for some reason? Note that times between process exiting can be up to a month.
  • We have the source for the main program, but would have problems running it inside a debugger since this is running at a customer, but maybe. We can't run it compiled for Debug though. Not at all, due to performance. This is live production.
  • Application is a normal WPF application without any child processes or any other inter-process communication. It does use a few third party devices e.g. libraries and drivers.
  • We have setup event handling of appdomain exceptions and application exceptions etc. None of these occur. The process exits without any indication of an exception occurring. It is a hard process exit.
  • We have suspected perhaps a third party driver being the source, but how? And how could we determine whether this was the case?

UPDATE 2: We use the nuget package TaskScheduler to setup the task via code. Note we do not set the ExecutionTimeLimit, which should therefore be Nothing and hence infinite.

using (TaskService m_service = new TaskService())
 {
     var task = m_service.NewTask();

     task.Principal.UserId = userId;
     task.Principal.LogonType = TaskLogonType.InteractiveToken;
     task.Principal.RunLevel = TaskRunLevel.Highest;

     task.Settings.Enabled = true;
     task.Settings.MultipleInstances = TaskInstancesPolicy.IgnoreNew;
     task.Settings.Hidden = false;

     // NOTICE: A subset of the following 4 settings will cause app to hang on Win10
     //task.Settings.AllowHardTerminate = true;
     //task.Settings.DisallowStartOnRemoteAppSession = false;
     //task.Settings.RunOnlyIfLoggedOn = true;

     var trigger = (LogonTrigger)task.Triggers.Add(new LogonTrigger());
     trigger.Enabled = true;
     trigger.UserId = userId;

     task.Actions.Add(new ExecAction(executableFilePath, arguments: null,
         workingDirectory: m_installDirectoryPath));

     if (!IsAdministrator())
     {
         var message = "Cannot register task with your current identity's permissions level.";
         m_logger.Error(message);
     }
     m_service.RootFolder.RegisterTaskDefinition(taskName, task, TaskCreation.Create,
         userId, password: null, logonType: TaskLogonType.InteractiveToken);
 }

UPDATE 3: Perhaps the above statement was wrong, the default in the TaskScheduler library seems to be 3 days or 72 hours.

//
// Summary:
//     Gets or sets the amount of time that is allowed to complete the task. By default,
//     a task will be stopped 72 hours after it starts to run.
//
// Remarks:
//     If a task is started on demand, the ExecutionTimeLimit setting is bypassed. Therefore,
//     a task that is started on demand will not be terminated if it exceeds the ExecutionTimeLimit.
[DefaultValue(typeof(TimeSpan), "3")]
public TimeSpan ExecutionTimeLimit { get; set; }

UPDATE 4: Only thing is we observe silent process exits after the process has been running for much longer than 3 days, e.g. 30 days so not sure it is this after all.

UPDATE 5: The longer than 3 days was not correctly observed, so after everything it is now clear this was due to incorrect settings for the task scheduler task. The wrong settings shown below:

Task Scheduler Wrong Settings

Correct settings being:

Task Scheduler Correct Settings

  • on stackoverflow a similar issue exists, but this seems more like a windows issue stackoverflow.com/questions/16992011/… – nietras Sep 3 at 11:22
  • Remark : Termination code 1067 means "ERROR_PROCESS_ABORTED - The process terminated unexpectedly". Not very helpful and tells nothing. – harrymc Sep 7 at 10:04
  • Remark 2 : It might be useful to separate the bundled services on svchost as described here. – harrymc Sep 7 at 10:10
  • @harrymc remark 2: Yes, we have already done that, waiting for next occurance, but good suggestion. – nietras Sep 7 at 10:28
  • @harrymc remark: yeah that is all we could find on 1067 too. Had been hoping it could indicate something... – nietras Sep 7 at 10:30
up vote 0 down vote accepted

It's just a theory, but maybe since it's started with a scheduled task that somehow the task scheduler still has "parent" control of the process after it's started? It would seem to me that there may be some condition that's reached or problem where the windows task scheduler is stopping the task. Perhaps you should try running it using the start command so that the application is started yet the "scheduled task" is allowed to complete, and therefore control of the application is released from windows task scheduler.

E.G. C:\Windows\System32\cmd.exe /c start "title" C:\Windows\System32\notepad.exe

(or whatever program you're running)

You have to run the cmd.exe because the start command is a built-in. Also, if you read the documentation for the start command you'll notice that the title argument is required. You can leave it as "Title" or set it to whatever you want.

  • Not sure why the down vote. That's how the task scheduler works. Try creating a task with no triggers that runs notepad.exe and start it manually. As long as the notepad.exe process is open the task will be seen as "running". By default, the task scheduler is configured to automatically close tasks that run longer than 3 days. Even if it's not that setting there may be some other condition that's met or a problem with the task scheduler that's closing the app. Once you end that notepad.exe process the task will revert to ready status. – apocalysque Sep 13 at 0:23
  • The intention is that answers be definitive solutions. It's great to have a theory. After you test it out and verify that it works, that would be the appropriate time to post it as an answer. From review. – fixer1234 Sep 13 at 3:34
  • It's only a theory because I don't have enough information to be able to tell what application this guy is running, let alone what that error would mean for the application. Obviously the guy doesn't want to say so I didn't ask. This is as good as it gets with the limited information provided. But since his logs specifically state that it's being terminated by the same process that's running the task scheduler I'd say that's a pretty strong correlation. The only way I can give this guy a definitive answer would be to sit at the box and debug it myself. I have tested the theory BTW. – apocalysque Sep 14 at 3:51
  • @apocalysque I will create the bounty again when we find out the exact cause of the issue, however, this might be a while, but if it was due to task scheduler I will of course give you the bounty. – nietras Sep 17 at 6:53
  • @apocalysque unfortunately I do not have enough points to create the bounty again, thought the points would go back. Anyway, you were right, it was wrong Task Scheduler settings see question update. – nietras Oct 3 at 7:10

I think that the information you supply in your post is all from after the fact.

For one, termination code 1067 just means "ERROR_PROCESS_ABORTED - The process terminated unexpectedly".

For another, the stack trace you gave is in my opinion just the processing of the event of your program termination, which is the thread that did the silent reporting, so has no information to add.

I think we will know more if the problem happens again after having separated the bundled services on svchost as described here. But again, designing svchost as the cause might be misleading.

In my own experience, the easiest sure way of finding the problem, requiring the full control of the product's sources, is compiling the product for debug and running it in the debugger, setting breakpoints on all the system termination routines. For C these are _exit and _abort, but there may be others.

If the program stops in the debugger, you will be able to see why by examining the call stack and global/local variables.

If you cannot use the debugger, we will need much more data about the program to be able to offer a meaningful opinion.

EDIT

Even compiling for production, you are at least able to create a symbols-file which can help in debug and analysis.

You should examine the Event Viewer - if the program crashes then there may be some info in there, and also in User-Mode Dump Files.

For better analyzing the crashes, you may parameter user dumps in Control Panel > System > Advanced system settings > Advanced tab > Startup and Recovery > Settings > Write debugging information, set it to "Complete memory dump" and OK.

You may use those dumps together with the symbols file in a debugger. You may do so on another computer, usually where you have compiled the executable, but there are some gotchas if you are using Visual Studio :

  • The dump and symbols file must have been generated by the same executable version
  • The dump must be located when being analyzed in exactly the same folder as when it was generated (C:\some-path\ etc.)
  • see updated question for more info, let me know if there are more questions. – nietras Sep 12 at 11:16
  • Added more info. And in what language is that executable written? – harrymc Sep 12 at 12:02
  • We have observed event viewer, possible logs, etc. There are no exceptions, I also mention this elsewhere. The process exits without any trace of exceptions or why. – nietras Sep 14 at 13:11
  • C# not that is matters I think. .NET 4.6. – nietras Sep 14 at 13:12
  • Just to know if you are using Visual Studio, but I added that part anyway. – harrymc Sep 14 at 13:25

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.