Can't connect to L2TP IPsec VPN from Windows 10 but it works with macOS High Sierra

Question

I am trying to connect from a Windows 10 client to an Ubiquiti EdgeRouter VPN, I went through the instructions on https://help.ubnt.com/hc/en-us/articles/204950294-EdgeRouter-L2TP-IPsec-VPN-Server but whenever I try to connect to the VPN I get the following:

Can't connect to The L2TP connection attempt failed because the security layer could not negotiate compatible parameters with the remote computer.

I did some research and changed the VPN security properties to allow Microsoft CHAP Version 2 (MS-CHAP v2) and also Challenge Handshake Authentication Protocol (CHAP) but it didn't seem to make a difference at all.

At the same time I have no problems whatsoever to connect from a macOS High Sierra client. So my guess is that there is something wrong on the client side (Windows VPN settings)? Can anybody help me out with this?

Answer 1

That error message probably means that there is some level of connection between Windows and the Ubiquiti but they failed to find a common encryption method. For phase1 (key exchange) Windows (version 1803) is proposing the following encryption methods (in this priority order):

1. SHA1 + AES-CBC-256 + ECP384
2. SHA1 + AES-CBC-128 + ECP256
3. SHA1 + AES-CBC-256 + MODP2048
4. SHA1 + 3DES-CBC + MODP2048
5. SHA1 + 3DES-CBC + MODP1024

For phase2:

1. SHA1 + AES-CBC-256
2. SHA1 + AES-CBC-128

Make sure the Ubiquiti can accept one of these proposals so try to set the phase1 and phase2 proposals on the Ubiquiti to something like this.

set vpn ipsec ike-group FOO0 lifetime 86400
set vpn ipsec ike-group FOO0 proposal 1 dh-group 14
set vpn ipsec ike-group FOO0 proposal 1 encryption aes256
set vpn ipsec ike-group FOO0 proposal 1 hash sha1
set vpn ipsec esp-group FOO0 lifetime 43200
set vpn ipsec esp-group FOO0 pfs disable
set vpn ipsec esp-group FOO0 proposal 1 encryption aes128
set vpn ipsec esp-group FOO0 proposal 1 hash sha1

I would also enable NAT-Traversal too:

set vpn ipsec nat-traversal enable

I do not really know Ubiquiti but maybe this helps.

Answer 2

Same issue here, but with a Cisco Meraki. Was working fine until my laptop upgraded to the 1803 build. I bet you have the same build. It was recently pushed out. Doesn't seem to matter what you do - there's something now wrong with L2TP.

If you're still on build 1709, then you need to add a firewall rule to allow ports 500, 1701, 4500 UDP/TCP both in and out.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/dd458955(v=ws.10)

Answer 3

IMHO this is a bug in Windows 10 built-in VPN client. I have two almost identical VPN servers running SoftEther. VPN connection to the first one had no problems since I remember (using SSTP). The second VPN connection on the same Windows PC to a VPN server almost identically configured and hosted by the same provider in the same network - when connecting via SSTP, I could read from server log that Windows client did not accept, or offer, MSCHAP v2 for authentication. Nothing could solve the problem until one day I tried to modify the Windows client to use L2TP/IPSec with PSK. It connected instantly. Then I just changed the configuration in Windows back to SSTP and it works like breeze ever since.