You seem to have an old style Ciso console cable. These are serial to serial. No VGA, Mini DP or USB is involved.
On one side the connector look like an RJ45 Ethhernet plug, but the network people always told me to call this 8P8C. It is a serial connector. On the other side you have a classic 9 pins connector.
If I am right it looks like this:
To connect it you need a serial port. These were very common on old boards, but these days they have mostly been replaced by USB. This leaves you with two options:
- But a USB to serial convertor. These are cheap (starting at €10) and often usefull.
- Find a motherboard with a serial connector. I assumed that these mostly died out but to my surprise I still found serial headers on my recent Z170 Skylake board.
Option 1 is probably the easiest.