PowerShell Remoting using CredSSP is broken

Question

I recently had PowerShell working perfectly with CredSSP, but now every time I try to establish a remoting session using CredSSP, I'm getting the following error:

Enter-PSSession : Connecting to remote server server01.contoso.com failed with the following error message : The WinRM client received an HTTP server error status (500), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic. At line:1 char:1 + Enter-PSSession -ComputerName server01.contoso.com -Credential $cred -Authentication C ... + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidArgument: (server01.contoso.com:String) [Enter-PSSession], PSRemotingTransportException + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

I've tried resetting the winrm service using winrm invoke restore winrm/config

I've tried disabling PowerShell remoting, disabling CredSSP (client and server), re-enabling PowerShell remoting, re-enabling CredSSP, disabling any GPOs related to configuring WinRM and credential delegation, and nothing has worked. Is there any way to dig into this further to figure out what's going on?

This is affecting all Windows Server 2012 systems in my lab environment, which are obviously running PowerShell v3.

In the client side, I'm seeing this in the Windows Remote Management event log: WSMan operation CreateShell failed, error code 2150859120

Configure Server

• Disable-PSRemoting -Force;
• Disable-WsmanCredssp -Role Client;
• Disable-WsmanCredssp -Role Server;
• Enable-PSRemoting -Force;
• Set-WSmanQuickConfig -UseSSL -Force;
• Enable-WsmanCredSSP -Role Server -Force;

Configure Client

• Disable-PSRemoting -Force;
• Disable-WsmanCredssp -Role Client;
• Disable-WsmanCredssp -Role Server;
• Enable-PSRemoting -Force;
• Set-WSmanQuickConfig -UseSSL -Force;
• Enable-WsmanCredssp -Role Client -DelegateComputer *.contoso.com -Force;

After configuring the client and server, run:

$cred = Get-Credential;
Enter-PSSession -ComputerName server.contoso.com -Credential $cred -Authentication CredSSP;

That reproduces the error consistently.

Answer 1

I think I may have found the issue.

I'd originally been running into problems with the HTTPS listener. I wanted to use an explicit IP but this is only available when using SSL. Setting up the SSL listener with the following:

Set-WSManQuickConfig -UseSSL -Force

or

winrm quickconfig -transport:https -Force

would configure the listener on the server but would still fail when connecting from the client with the -UseSSL parameter.

I gave up on the IPs and returned to using machine names. I left the attempts to set HTTPS in the script but ran into the issues you were having with the 500 responses when using Credssp for authentication.

Finally, I decided to try one thing at a time. As soon as I removed the HTTPS settings, things worked!

My full script looks like this:

# Disable/revoke winrm/remoting
Start-Service winrm
winrm invoke restore winrm/config

Disable-PSRemoting -Force
Disable-WSManCredSSP -Role Client
Disable-WSManCredSSP -Role Server
Stop-Service winrm

# Enable remoting
Enable-PSRemoting -Force
Enable-WSManCredSSP -Role Server -Force
Enable-WSManCredSSP -Role Client -DelegateComputer "*.mydomain.com" -Force
winrm enumerate winrm/config/listener

Set-Item WSMan:\localhost\Client\TrustedHosts "*.mydomain.com" -Force

It's certainly not ideal,, but I hope it helps.

Answer 2

Quick Summary

For anyone else searching how to systematically resolve this problem it basically comes down to changing a file ACE accessed by lsass.exe impersonating as NT AUTHORITY\NETWORK SERVICE.

Details

1) Getting this error indicates WinRM for remoting is setup correctly, otherwise a non-500 http error will be encountered.

2) My problem was isolated to using localhost, testing both the client/server portions on the same machine using the following PS command:

$cred = Get-Credential 
Enter-PSSession -ComputerName localhost -Credential $cred -Authentication Credssp

3) I generated a self-signed certificate with the following PS command:
Note: substitute the actual hostname for <hostname> without the angle brackets <>. If the machine is part of a domain add the FQDN to the list.

$todaydt = Get-Date
$3years = $todaydt.AddYears(3)
New-SelfSignedCertificate -dnsname <hostname>, localhost -notafter $3years -CertStoreLocation cert:\LocalMachine\My

4) The above certificate gets added with a private key to the OS certificate store on the local computer under:

\Personal\Certificates

Using certlm.msc export the certificate (with private key) and then import it under - yes you will have two copies:

\Trusted Root Certificate Authorities\Certificates

5) The New-SelfSignedCertificate will output a Thumbprint, make sure this value is stored as the CertificateThumbprint in the following two WinRM locations (see below about how to easily update WinRM values via the registry)

winrm e winrm/config/listener
Look for the CertificateThumbprint under the HTTPS transport
[HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN\Listener\*+HTTPS\certThumbprint]

winrm get winrm/config/service
[HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN\Service\credssp_thumbprint]

6) Enable PS tracing to capture the exact error message thrown by the server:

Enable-PSWsmanCombinedTrace
Enter-PSSession -ComputerName localhost -Credential $cred -Authentication Credssp
Disable-PSWsmanCombinedTrace

7) Open the resulting trace file with the Event Viewer:

C:\Windows\System32\WindowsPowerShell\v1.0\Traces\PSTrace.etl

8) In this case the error was reported as a Server Response handling:

Sending HTTP error back to the client due to a transport failure.
The HTTP status code is 500
The error code is 2148074253

This can also be represented as 0x8009030D or SEC_E_UNKNOWN_CREDENTIALS

9) Use Sysinternals Process Monitor to find the exact file(s) getting ACCESS DENIED for the process lsass.exe

C:\ProgramData\Microsoft\Crypto\Keys\
C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

10) Use the following icacls command to update the ACE on the file(s)

icacls C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\<###> /t /c /grant "NT AUTHORITY\NETWORK SERVICE:(R)"
icacls C:\ProgramData\Microsoft\Crypto\Keys\<###> /t /c /grant "NT AUTHORITY\NETWORK SERVICE:(R)"

11) Everything should be good to go

Enter-PSSession -ComputerName localhost -Credential $cred -Authentication Credssp
Enter-PSSession -ComputerName localhost -Credential $cred -Authentication Credssp -UseSSL

Handy Commands

(A) To see the current WinRM configuration information

winrm get winrm/config
winrm e winrm/config/listener

The above configuration information is stored in the registry and the values can easily be modified on the fly from the location below and seen immediately updated in the above commands.

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WSMAN

(B) Enable PS remoting and bypass "failures" when the network is "Public"

Enable-PSRemoting -SkipNetworkProfileCheck -Force

(C) Examine the current CredSSP configuration

Get-WSManCredSSP

(D) Enable CredSSP for both the server and client roles

Enable-WSManCredSSP -Role "Server"
Enable-WSManCredSSP -Role "Client" -DelegateComputer "localhost", "127.0.0.1", "<hostname>", "<FQDN>"