I currently configured openVPN and shadowsocks(based on socks5) on a server. I am confused what is the difference between two and when should I use one over other.
Although they can perform similar things, under the hood OpenVPN and Shadowsocks are entirely different beasts.
OpenVPN is a VPN server. Shadowsocks is a Proxy.
A proxy can (optionally using encryption) ask another computer to get information for you. Generally the intermediate system is actually doing some processing of the packets. This limits the programs which can run on top of it.
A VPN encrypts packets sent between points - in the case of a VPN, other then stripping encryption and forwarding the raw packet, the VPN server does modify the packets. (*a VPN server may additionally deploy NAT which will rewrite the source or target address of the packets). VPN's are more flexible. An application needs to be built to work with SOCKS.
VPN servers are generally (but not always) easier for governments to detect and block, as they generally speak their own protocol to make them more general purpose. Applications do not need to modification to work with a VPN. Also, the nature of a VPN is such that long sessions can be detected, even when web browsing - while a proxy server will open and close the connection like a web page.
Thus the answer to which to use depends on your purpose in using one. If "flying under the radar" while using a limited set of protocols/programs is more valuable, use SOCKS. If greater flexibility of protocols or more control of the underlying delivery mechanism is required, use OpenVPN.
Another way to think of it is that a VPN overlays a "virtual network" at (close to) an OS level on top of part of an actual actual network, while a proxy creates applications to talk to each other in an encrypted way without virtualising the underlying network.
"Shadowsocks is based on a technique called proxying. Proxying grew popular in China during the early days of the Great Firewall—before it was truly “great.” In this setup, before connecting to the wider internet, you first connect to a computer other than your own. This other computer is called a “proxy server.” When you use a proxy, all your traffic is routed first through the proxy server, which could be located anywhere. So even if you’re in China, your proxy server in Australia can freely connect to Google, Facebook, and the like.
But the Great Firewall has since grown more powerful. Nowadays, even if you have a proxy server in Australia, the Great Firewall can identify and block traffic it doesn’t like from that server. It still knows you are requesting packets from Google—you’re just using a bit of an odd route for it. That’s where Shadowsocks comes in. It creates an encrypted connection between the Shadowsocks client on your local computer and the one running on your proxy server, using an open-source internet protocol called SOCKS5.
How is this different from a VPN? VPNs also work by rerouting and encrypting data. But most people who use them in China use one of a few large service providers. That makes it easy for the government to identify those providers and then block traffic from them. And VPNs usually rely on one of a few popular internet protocols, which tell computers how to talk to each other over the web. Chinese censors have been able to use machine learning to find “fingerprints” that identify traffic from VPNs using these protocols. These tactics don’t work so well on Shadowsocks, since it is a less centralized system."
All information is from this site: https://qz.com/1072701/meet-shadowsocks-the-underground-tool-that-chinas-coders-use-to-blast-through-the-great-firewall/