Preliminary notes
This answer is Unix-centric. ssh can be used in Windows. In Windows some concepts mentioned in this answer may not apply. Example commands use quoting and pathnames from the Unix world.
I use OpenSSH. Other implementations may miss some features this answer uses.
About SSH tunnels
This other answer is technically correct but it does not really address your problem. It's true the SSH protocol provides encryption and authentication. The client and the server authenticate to each other when the SSH connection starts. All data travelling through the connection is encrypted, including tunneled data; but…
Access to a tunnel is not restricted by SSH in any SSH-specific way. If the listening end is a TCP port then restrictions specific to networking will apply: interface (e.g. the local interface is not available from the outside), firewall etc. If the listening end is a Unix socket then restrictions specific to file access will apply (because the socket is a file): mode, ownership, etc. (note: manuals warn that not all operating systems honor the file mode on Unix-domain socket files; still creating a socket in a directory with restricted access should restrict access in any system).
If one can open the listening end then he or she can use the tunnel. There will be no prompt or anything injected by SSH. By design a tunnel created with ssh -L or ssh -R relays data from one end to the other (bidirectionally) as-is; it's meant to be transparent. This means when you connect to some_address:port there's no difference if the server is there on the machine with some_address, or it's behind one or more tunnels on a different machine; the server gets what you send and you get what the server sends.
If your RC uses ssh -R to tunnel the port B of VS to any address:port RC can access (in general address:port may or may not belong to RC) then connecting to port B of VS is like connecting to address:port from RC. If the listening port B of VS can be accessed from the Internet then anyone can reach address:port as if they were on RC. The transparency of the tunnel allows this. If the protocol being tunneled allows authentication, use it maybe. In general there may be no protocol, you may want to send an arbitrary stream, the tunnel will relay it as-is.
In your case maybe port B of VS cannot be reached from the Internet, but port A can and it is redirected to B. One way or another there is some exposed port that allows anyone connect to address:port as if they were on RC.
Improving security in general case
There are few methods of improving the setup and making it less open to the public:
A firewall on VS may be configured to deny access to the exposed port from anywhere but LC. This requires LC to use a static IP address from which its communication gets to VS. If LC is behind NAT then VS will not be able to reliably tell apart LC from another host behind the same NAT. Setting firewall rules on VS requires administrative access to VS. You probably don't want to (or cannot) use this method.
RC, when connecting with ssh -R, can make the SSH server on VS bind to the loopback interface of VS. Assuming the chosen port is 5555, it will be like ssh -R 127.0.0.1:5555:address:port …. Then VS should not redirect any exposed port to its end of the tunnel. To use the tunnel one needs to connect to 127.0.0.1:5555 (or similar address) from within VS.
To connect from LC you need to find a way to reach 127.0.0.1:port of VS as if you were on VS. This is simple, if only you have SSH access to VS. Use ssh -L 127.0.0.1:7777:127.0.0.1:5555 … on LC to tunnel port 7777 of LC to the right port on VS. Then on LC connect to 127.0.0.1:7777. This connection will use the two chained tunnels and it will ultimately reach address:port from RC as if you were on RC.
The method requires SSH access to VS from LC. Random guys from the Internet will not be able to use the tunnel. Users of LC will, if they connect to 127.0.0.1:7777; this includes users other than you. On a private computer this is usually not a problem (see the notes section way below). However users of VS will be able to use the tunnel if they connect to 127.0.0.1:5555 inside VS. In a multi-user environment this may be a problem.
To secure your tunnel from other users of VS, modify the previous method, so instead of using a TCP port on VS it uses a Unix socket. On RC you need to connect to VS with ssh -R /path/to/socket:address:port …; on LC you need to connect to VS with ssh -L 7777:/path/to/socket …. Use arbitrary /path/to/ directory where the user (I mean the user RC connects as, i.e. the user RC specifies when ssh-ing to VS) can create files in VS. There are at least two problems:
ssh -R from RC will by default create the socket with such ownership and mode, so only the user RC connects as (or VS's root) can use it. In some systems the socket permissions are ignored (restrict access to /path/to/ then) but in Linux they work (see Pathname socket ownership and permissions in man 7 unix). This will secure your tunnel from other users of VS, what is exactly what we want. If LC connects to VS as the same user as RC then it's perfect. If from LC you act as another user on VS then the socket will have to be made available to you. RC will need to run setfacl in VS after creating the socket. This can be done by shell code supplied when RC does ssh -R (and the code should end with exec sleep 2147483647 to prevent the connection from terminating, this number should be safe and enough).
If /path/to/socket already exists on VS then it must be removed before ssh -R can work. An automatic solution exists, it requires StreamLocalBindUnlink to be yes on the server, in your case on VS. If you cannot configure VS this way then RC should ssh … 'rm /path/to/socket' first, before ssh -R. StreamLocalBindUnlink exists as an option for ssh but I think it applies to sockets created on the client side (we don't use such sockets here); some say it's a bug.
Using sockets instead of TCP ports not only allows you to keep other users on VS away from your tunnel; it also prevents them from tricking you into using the wrong tunnel. Imagine RC uses ssh -R to create a tunnel only to find out the port it wants to use on VS is already taken. Some other user is using it, maybe as a tunnel. Maybe it leads to the same type of service you expect from RC. You don't know RC failed, you connect from LC as usual and send your data. From the point of view of the other user you are a random guy who uses their setup uninvited; or a victim who sends your sensitive data to them. This cannot happen if you use Unix sockets properly.
Note in the above cases you don't really need SSH access to RC. I mean whatever code RC should run, it can be set up once and automated by RC's administrator who may not be you. RC may not even run sshd (a server); ssh (client application) is enough for it. OTOH to chain the tunnels you need SSH access from LC to VS.
SSH access to RC gives you more options.
Specific case: tunneling SSH via SSH
I assume you cannot directly reach RC from LC (otherwise VS would not be needed in the first place). If it's the only thing that prevents you from ssh-ing from LC to RC, i.e. if an SSH server runs on RC and you have a key or a password to connect, then you can build tunnels in yet another way.
First you need to be able to access the SSH server of RC. This is easy: take the general case (elaborated above) of address:port and apply it for 127.0.0.1:22.
Even if RC does ssh -R '*:5555:127.0.0.1:22' … and therefore requests VS to listen on all interfaces (including exposed ones) or if VS redirects some exposed port to the tunnel, the setup is quite secure. This is because anyone from the Internet who connects to the listening end of the tunnel will reach the SSH server of RC which requires authentication anyway.
I know tunneling SSH is not your goal. The point is now you can create any TCP tunnel you want between LC and RC by invoking ssh on LC. You can create many tunnels. E.g. to reach from LC some arbitrary address:port as if you were on RC, invoke on LC:
ssh -p 5555 -NL 127.0.0.1:1234:address:port userRC@VS
and on LC use 127.0.0.1:1234 in any program you want to connect to address:port.
Notes:
The SSH connection that tunnels your data is itself tunneled between VS and RC. So it's a tunnel in a tunnel.
userRC@VS means you need to use the address of VS but the username (and credentials) for RC. We're using the address of VS only because it's the listening end of the tunnel that leads to RC. You don't need SSH access to VS.
It may happen RC cannot create its tunnel because somebody else has taken the chosen port on VS. We noted this possibility for the general case, it was insecure. Now it's not that bad. If this happens and you (from LC) try to connect to what you think is the SSH server of RC, you will be able to tell it's not RC, if only you're paying attention to fingerprints. ssh with its known_hosts file will try to help, although it may not tell apart VS:22 from VS:5555, so you may get false warnings. Anyway in principle you will be able to tell.
This means a "hijacker" cannot trick you if you're careful. Still he or she can prevent you from reaching RC. The method using Unix sockets to create chained tunnels will come handy (note it requires SSH access from LC to VS). Again, take the general case (elaborated above) of address:port and apply it for 127.0.0.1:22. The method involves ssh -R on RC and ssh -L on LC. Then you are able to reach the SSH server of RC by connecting to 127.0.0.1:7777 on LC. Finally an additional ssh command on LC in a form of:
ssh -p 7777 -NL 127.0.0.1:1234:address:port [email protected]
will allow you to create a tunnel to address:port (seen from RC) locally available as 127.0.0.1:1234. It will be a tunnel in the chained tunnels.
Specific case: SOCKS
You wrote:
I'd like to run a SOCKS server on a remote client (RC) […]
If a SOCKS server can be reached via address:port from RC then to connect from LC you can use any of the above methods that are not specific to SOCKS.
However if you can access the SSH server of RC from LC then you don't need a standalone SOCKS server. There is ssh -D, it's somewhat similar to ssh -L, it listens on a local port. Instead of forwarding the port to some arbitrary address:port on the remote side, ssh -D creates a SOCKS server for you, as if the port was tunneled to some SOCKS server running beside the SSH server on the remote machine. It's a built-in SOCKS server to use with ssh.
To use it as if a standalone SOCKS server was running on RC, you need SSH access to RC. That's why the previous section of this answer concentrated on reaching the SSH server of RC from LC first.
If you could reach RC directly (without VS), you would use ssh -D on LC this way:
ssh -ND 127.0.0.1:8888 userRC@RC
and you would configure your browser (or whatever) to use 127.0.0.1:8888 as the SOCKS proxy. With tunneling via VS the situation is like in the previous section, except you want -D 127.0.0.1:8888 instead of -L 127.0.0.1:1234:address:port. Example:
ssh -p 5555 -ND 127.0.0.1:8888 userRC@VS
In general this is what I would do if I were you. I would set up SSH access from LC to RC, then I would use ssh -D on LC.
Less known is the fact ssh -R can act as SOCKS proxy. If the tunnel RC creates was built without specifying address:port, i.e. if the command was like ssh -R 127.0.0.1:5555 …, then you could use 127.0.0.1:5555 of VS to reach SOCKS proxy running on RC (as if a standalone SOCKS proxy was on address:port there and the tunnel lead to it). So an administrator of RC can give you SOCKS proxy without running a standalone SOCKS server or even an SSH server, ssh is enough. From LC you would use it like in the general case. The difference is ssh -R acting as SOCKS proxy cannot bind to a Unix socket (at least not at the time I'm writing this), it has to be a TCP port. Firmly restricting access by using a Unix socket instead is therefore impossible.
Notes
ssh and sshd are highly configurable. There are many options (see man 5 ssh_config and man 5 sshd_config). The same simple ssh command may work differently depending on options in the files. E.g. when I say "ssh -R from RC will by default create the socket with ownership and mode so only the user RC connects as […] can use it", do not read "by default" as "when there are no further options to ssh in the command line". The relevant option is StreamLocalBindMask on the server (in sshd_config), I think, if it's specified then "by default" does not apply. There are more options that can interfere with what you want to do, e.g. AllowTcpForwarding.
Creating a SOCKS proxy with ssh is explained in this answer of mine: How to create a SOCKS proxy with ssh? In the context of the linked answer your LC is A and B (A=B case), your RC is C. Your VS sits between B and C, the linked answer does not consider such node. Once you move VS out of the problem by making ssh from LC to RC possible, the linked answer can be useful.
Even the most secure setup we considered in this answer allows other users of LC to use the ultimate tunnel. If this is a problem and if the program that is going to use the tunnel can use a Unix socket instead of a port, consider a socket in LC. If the program must use a TCP port, I guess a separate network namespace can help; but setting it up is not trivial, I won't elaborate. If you want security without excessive effort, being a sole user of LC helps.
Being a sole user of VS also helps. After all I wrote it should be clear why it is so.
UDP
Would it be possible to extend this to UDP connections as well?
When you can reach RC from LC via ssh, you can create more than one tunnel. But not UDP, not as easily. There is no support (why?). Tricks that read UDP, write to TCP, tunnel TCP and "translate" back to UDP (example) are flawed, do not trust them. UDP preserves message boundaries, TCP does not, this is where the flaw comes from. This answer mentioning ssh -w seems good (at least not flawed in theory) but I haven't tested. A full-scale VPN to which you connect using a TCP tunnel is a possibility, although it may be slow.
