Short answer: Yes, all of the above.... almost.
First off, let's clarify the word "server". Server can refer to both the machine (virtual or physical), and to the individual processes handling client requests. for example apache (http) or named (dns). In this answer I will use "server" to refer to the processes, and "machine" to refer to, in your case, the VPS.
A private nameserver generally means a DNS server - the server that translates between IP addresses (for example 123.123.123.123) and hostnames (example.com). This DNS server can either be a slave used just by you, or what is more applicable, a zone master. In general, what you do is set up your DNS server such that it acts as master for a domain name you control. Let's call this domain name example.com.
The process of operation is roughly this:
- Peter the PC eater tries to go to the website www.example.com
- Peters computer makes a DNS lookup for www.example.com by asking whatever DNS server Peter gets from his ISP.
- Said DNS server doesn't know, so it refers Peters PC up the chain. Eventually Peters PC reaches the root DNS server, which refers Peters PC to the master for the .com zone. With whoever you purchased the domain from, you should have set up that the DNS server for your domain is located on the IP of your VPS, and this is the reply that Peter eventually gets. While it sounds fairly tedious, a DNS request is usually very small, so this rarely takes more than a few miliseconds on a decent connection.
- Peter, knowing the IP of the DNS server for example.com now makes a DNS request to said IP. DNS requests are done via port 53, and on that you (should) have a DNS server listening for requests. If set up properly, the DNS server responds with the IP of your webserver.
- Peters PC then connects its webbrowser to the IP returned earlier, and if all is set up correctly, loads the website as intended.
It does not matter that your machine run both a DNS server and a webserver. These run on different ports (port 53 and 80 respectively). Neither does it matter that all of the lookups and connections end up on the same IP.
I use the same machine for many services:
- DNS
- Web
- Mail
- SSH
- Loads more
Why this works is that they all run on different ports. Every server that has an IP has a port associated with it. Ports range from 1 to 65535, and many of them are reserved. In particular the ones below 1024. See here for a long list. Of particular interest to you in this case are port 53 and 80, as briefly mentioned earlier on.
While I do not know any details about your VPS, I can give you some pointers on what to read up on to get you started:
- DNS servers - for example
named
- Web servers - for example apache, also just called
httpd
- IP addressing and port numbering
- Basic socket operation