I've been for some days on this problem and I'm starting to run out of ideas on this one (I'm not a network expert).
I'm working on a project using OpenVPN to connect to a random external VPN provider. This project is working for month on a private dedicated server without any problem. I'm trying to switch to cloud using Kubernetes. Everything works fine... but not the VPN connection. I use a NodeJS program to start and manage VPN Connection. Whenever the connection goes down, the program tries to restart openvpn until it works (every 30 seconds). The problem is that is "restart" feature works great on dedicated server with no visible downtime, but on cloud version, the downtime goes from 5 secondes to about 10 Minutes. I see no pattern and it seems at least really random.
I've been digging for some days now and I can see the "root cause": whenever I try to restart the VPN connection, I get sometimes a "RTNETLINK answers: Network unreachable" when trying to add some routes and I don't know why for now. Here are the logs I get everytime the connection fails:
[debug]: stdout: Fri Aug 30 13:08:16 2019 /sbin/ip route add [error]: OpenVPN error: RTNETLINK answers: File exists [debug]: stdout: Fri Aug 30 13:08:16 2019 ERROR: Linux route add command failed: external program exited with error status: 2 /sbin/ip route add 0.0.0.0/1 via 172.21.20.1 [error]: OpenVPN error: RTNETLINK answers: Network unreachable [debug]: stdout: Fri Aug 30 13:08:16 2019 ERROR: Linux route add command failed: external program exited with error status: 2 [debug]: stdout: Fri Aug 30 13:08:16 2019 /sbin/ip route add 126.96.36.199/1 via 172.21.20.1 [error]: OpenVPN error: RTNETLINK answers: Network unreachable Initialization Sequence Completed
I now that connection fails because I have an automatic check that gets my external IP and checks if it is my real one or one of the VPN provider.
From what I've seen, the first error is not a problem because when its the only that occurs, VPN connection is ok. But the "Network unreachable" is the sign that something went wrong. But then, after some retries every 30 secondes, it finally connects.
I first thought that I was not killing properly openvpn process thus not freeing properly some resources, but It doesn't seem to be it because just running the same command :
openvpn --config conf.ovpn --dhcp-option DNS 188.8.131.52 --auth-user-pass /path/auth.txt
Various times (and killing with a Ctrl+C), it sometimes passes and sometimes not. And my project is at risk if I can't manage to have a stable connection inside the cloud (I have more features to come and can't stay on a dedicated server).
Here are also the push from the server:
PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1 bypass-dhcp,dhcp-option DNS 198.18.0.1,dhcp-option DNS 198.18.0.2,rcvbuf 493216,sndbuf 493216,explicit-exit-notify 5,comp-lzo no,route-gateway 172.21.22.1,topology subnet,ping 20,ping-restart 40,ifconfig 172.21.23.169 255.255.254.0,peer-id 5,cipher AES-256-GCM'
And the OpenVPN config file:
client dev tun proto udp remote XXX.XXX.com 443 resolv-retry infinite nobind persist-key persist-tun persist-remote-ip ca XXX.crt verify-x509-name XXX.com name auth-user-pass comp-lzo verb 3 auth SHA256 cipher AES-256-CBC keysize 256 tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-DSS-WITH-AES-256-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA
Again, all this works properly but not on the kubernetes environment. So it must have something to do with it or my provider?
Any help to debug this would be highly appreciated.