- I know I need to have domain, so how do I create free domain for testing?
You may already have a domain name provided by your ISP. If you are expecting to accept email from the Internet, you will want to forward port 25 from the router to your server. You could also use a Dynamic DNS service, many of which have available sub-domains you can register with. Don't expect to be able to reliably send directly to the Internet with either of these options.
I don't think I need to setup DNS server on my virtual SMTP server(windows server 2008), correct me if I'm wrong.
You will need a DNS server, but it is common to use your DNS registrars servers if you have a small domain.
If you are doing SPAM filtering you may want a caching DNS server for the spam filter to use. This does not need to be the DNS server for any domain.
What extra configuration/services I need to start?
You only need the SMTP service on the server which is providing MX (Mail Exchange) or MTA (Mail Transfer Agent) facilities.
Any good source from which I can refer to for setting up SMTP server?
My rant on Running a Mail Server may be a good place to start. It points out commonly made mistakes that I have seen even large organizations making.
If you are sending email to the Internet, I recommend your server act as a smart host and have it use your ISP's relay server to send email. You should also look at SPF (Sender Policy Framework) and DKIM (Domain Keys Identified Mail). My published Email Policy ends with a list of resources.
Accepting email on a dynamic IP address may result in email being delivered to some other server or rejected. You should also be ready to deal with incoming Spam. Read the RFCs related to SMTP and other electronic mail subjects. Many are listed in my Email Policy document.