Silent Process Exit: process '?' was terminated by the process 'C:\Windows\System32\svchost.exe' with termination code 1067

Question

We have a serious issue with a C# application being terminated silently at random and infrequent points in time on a Windows 10 32-bit installation. E.g. it might be a month between occurrences. Or sometimes just a day.

Basic system specifications:

Microsoft Windows 10 Enterprise 2016 LTSB
Version 10.0.14393 Build 14393
32-bit

Using https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-and-clearing-flags-for-silent-process-exit we have configured silent process exit monitoring. And we finally have a few samples of this:

The process 'APPLICATIONPATH\APPLICATIONNAME.exe' was terminated by 
the process 'C:\Windows\System32\svchost.exe' with termination code 1067. 
The creation time for the exiting process was 0x01d43bd8689073eb.

Looking at the dumps for this, which was setup for the monitoring we got a process ID for the svchost. This service was still running at the system, and it shows the following list of services:

Which seems to be a list of "netsvcs" for Windows. Opening the dump from the svchost.exe and looking at this a single thread was found with an interesting call stack:

ntdll.dll!_KiFastSystemCallRet@0 ()
ntdll.dll!_NtWaitForSingleObject@12 ()
ntdll.dll!RtlReportSilentProcessExit()
KERNELBASE.dll!TerminateProcess()
ubpm.dll!_UbpmpTerminateProcessCallback@12 ()
ubpm.dll!UbpmUtilsTimerCallback()
ntdll.dll!TppTimerpExecuteCallback()
ntdll.dll!TppWorkerThread()
kernel32.dll!@BaseThreadInitThunk@12 ()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart@8 ()

UBPM is the Unified Background Process Manager. But how can this be terminating our application? And why? And what does the termination code 1067tell us?

Below is the log entry from Silent Process Monitoring:

Log Name:      Application
Source:        Microsoft-Windows-ProcessExitMonitor
Date:          2018-08-31 15:26:09
Event ID:      3001
Task Category: None
Level:         Information
Keywords:      Classic
User:          SYSTEM
Computer:      PC
Description:
The process 'APPLICATIONPATH\APPLICATIONNAME.exe' was terminated by the process 'C:\Windows\System32\svchost.exe' with termination code 1067. The creation time for the exiting process was 0x01d43ed2aee892ab.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-ProcessExitMonitor" Guid="{FD771D53-8492-4057-8E35-8C02813AF49B}" EventSourceName="Process Exit Monitor" />
    <EventID Qualifiers="16384">3001</EventID>
    <Version>0</Version>
    <Level>4</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2018-08-31T13:26:09.988216500Z" />
    <EventRecordID>4853</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>PC</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData Name="EVENT_PROCESSTERMINATION_CROSSPROCESS">
    <Data Name="param1">APPLICATIONPATH\APPLICATIONNAME.exe</Data>
    <Data Name="param2">C:\Windows\System32\svchost.exe</Data>
    <Data Name="param3">1067</Data>
    <Data Name="param4">01d43ed2aee892ab</Data>
  </EventData>
</Event>

NOTES: The PC is not being shut down at the moment the app terminates nor are there any other indications in event logs as to why the process was terminated.

UPDATE 1: Here a few extra details (trying to answer as many as in comments):

• Process is (sometimes) started via TaskScheduler when Windows is started yes. Other times by user. Not entirely sure problem only occurs when started via TaskScheduler. But interesting point? Could windows kill a task for some reason? Note that times between process exiting can be up to a month.
• We have the source for the main program, but would have problems running it inside a debugger since this is running at a customer, but maybe. We can't run it compiled for Debug though. Not at all, due to performance. This is live production.
• Application is a normal WPF application without any child processes or any other inter-process communication. It does use a few third party devices e.g. libraries and drivers.
• We have setup event handling of appdomain exceptions and application exceptions etc. None of these occur. The process exits without any indication of an exception occurring. It is a hard process exit.
• We have suspected perhaps a third party driver being the source, but how? And how could we determine whether this was the case?

UPDATE 2: We use the nuget package TaskScheduler to setup the task via code. Note we do not set the ExecutionTimeLimit, which should therefore be Nothing and hence infinite.

using (TaskService m_service = new TaskService())
 {
     var task = m_service.NewTask();

     task.Principal.UserId = userId;
     task.Principal.LogonType = TaskLogonType.InteractiveToken;
     task.Principal.RunLevel = TaskRunLevel.Highest;

     task.Settings.Enabled = true;
     task.Settings.MultipleInstances = TaskInstancesPolicy.IgnoreNew;
     task.Settings.Hidden = false;

     // NOTICE: A subset of the following 4 settings will cause app to hang on Win10
     //task.Settings.AllowHardTerminate = true;
     //task.Settings.DisallowStartOnRemoteAppSession = false;
     //task.Settings.RunOnlyIfLoggedOn = true;

     var trigger = (LogonTrigger)task.Triggers.Add(new LogonTrigger());
     trigger.Enabled = true;
     trigger.UserId = userId;

     task.Actions.Add(new ExecAction(executableFilePath, arguments: null,
         workingDirectory: m_installDirectoryPath));

     if (!IsAdministrator())
     {
         var message = "Cannot register task with your current identity's permissions level.";
         m_logger.Error(message);
     }
     m_service.RootFolder.RegisterTaskDefinition(taskName, task, TaskCreation.Create,
         userId, password: null, logonType: TaskLogonType.InteractiveToken);
 }

UPDATE 3: Perhaps the above statement was wrong, the default in the TaskScheduler library seems to be 3 days or 72 hours.

//
// Summary:
//     Gets or sets the amount of time that is allowed to complete the task. By default,
//     a task will be stopped 72 hours after it starts to run.
//
// Remarks:
//     If a task is started on demand, the ExecutionTimeLimit setting is bypassed. Therefore,
//     a task that is started on demand will not be terminated if it exceeds the ExecutionTimeLimit.
[DefaultValue(typeof(TimeSpan), "3")]
public TimeSpan ExecutionTimeLimit { get; set; }

UPDATE 4: Only thing is we observe silent process exits after the process has been running for much longer than 3 days, e.g. 30 days so not sure it is this after all.

UPDATE 5: The longer than 3 days was not correctly observed, so after everything it is now clear this was due to incorrect settings for the task scheduler task. The wrong settings shown below:

Correct settings being:

Answer 1

It's just a theory, but maybe since it's started with a scheduled task that somehow the task scheduler still has "parent" control of the process after it's started? It would seem to me that there may be some condition that's reached or problem where the windows task scheduler is stopping the task. Perhaps you should try running it using the start command so that the application is started yet the "scheduled task" is allowed to complete, and therefore control of the application is released from windows task scheduler.

E.G. C:\Windows\System32\cmd.exe /c start "title" C:\Windows\System32\notepad.exe

(or whatever program you're running)

You have to run the cmd.exe because the start command is a built-in. Also, if you read the documentation for the start command you'll notice that the title argument is required. You can leave it as "Title" or set it to whatever you want.

Answer 2

I think that the information you supply in your post is all from after the fact.

For one, termination code 1067 just means "ERROR_PROCESS_ABORTED - The process terminated unexpectedly".

For another, the stack trace you gave is in my opinion just the processing of the event of your program termination, which is the thread that did the silent reporting, so has no information to add.

I think we will know more if the problem happens again after having separated the bundled services on svchost as described here. But again, designing svchost as the cause might be misleading.

In my own experience, the easiest sure way of finding the problem, requiring the full control of the product's sources, is compiling the product for debug and running it in the debugger, setting breakpoints on all the system termination routines. For C these are _exit and _abort, but there may be others.

If the program stops in the debugger, you will be able to see why by examining the call stack and global/local variables.

If you cannot use the debugger, we will need much more data about the program to be able to offer a meaningful opinion.

EDIT

Even compiling for production, you are at least able to create a symbols-file which can help in debug and analysis.

You should examine the Event Viewer - if the program crashes then there may be some info in there, and also in User-Mode Dump Files.

For better analyzing the crashes, you may parameter user dumps in Control Panel > System > Advanced system settings > Advanced tab > Startup and Recovery > Settings > Write debugging information, set it to "Complete memory dump" and OK.

You may use those dumps together with the symbols file in a debugger. You may do so on another computer, usually where you have compiled the executable, but there are some gotchas if you are using Visual Studio :

• The dump and symbols file must have been generated by the same executable version
• The dump must be located when being analyzed in exactly the same folder as when it was generated (C:\some-path\ etc.)