As @sysadmin1138 commented, spam and filtering is going to be the biggest obstacle. I would second the recommendation of Zimbra, but suggest that if you can spare another box (or some virtual space on the same box) set up a Mailcleaner mail gateway/filter to receive, scrub, and relay mail to the Zimbra server.
Mailcleaner is an Exim-based filtering solution with a web frontend that nicely wraps up lots of great open-source tools. It's no harder to set up than Zimbra or Scalix, and enabling even just some of the SMTP-time RBLs will cut down on your spam volume significantly. Other important stuff like Razor, Pyzor, and DCC can be configured to scrub further. ClamAV is built in as well.
Both Zimbra and Mailcleaner play well with virtualization options like VMWare (who now owns Zimbra) and KVM -- IMO a Linux box running Mailcleaner and Zimbra guests on KVM would be a great, no-cost home/personal email server setup.
Lastly, if you plan to run this on a residential or home Internet connection you'll likely need to look into dynamic DNS to set up forward and reverse mail records unless you have a static IP. You should also contact your ISP or take a gander at their terms of service to determine if they will have an issue with hosting your own email server.