I only use a browser to surf the Internet, and nothing else.
Then you must at all times keep that browser updated, along with any other Internet-facing code such as browser plug-ins. Keeping the Flash, Java, Windows Media Player, QuickTime, Real, Office and Acrobat plug-ins all up-to-date is a pain, so just don't install the ones you don't need. (You probably don't need any of those except Flash, and Flashblock can limit the damage from that.)
Otherwise you're going to get hit by browser exploits and be infected. Avoiding dodgy porn sites won't help you; even if you only view “trusted” sites, they can be hacked or include adverts from a malicious source. There are many, many examples of big mainstream sites leading to browser exploits.
And here's the point: today's antivirus isn't going to protect you. There is an enormous amount of exploit code and payload out there and it's being updated every day to keep it out of antivirus software definitions. This is a huge industry and antivirus is not at all keeping up. New web sploits I submit to virustotal are often seeing single-digit detection rates.
If you get hit, you're likely to have a number of different payloads installed, including stealthy ones like rootkits. The chances of your antivirus software correctly detecting and removing all of them are slim. So if you want to be safe, you'd have to reinstall your OS. At which point having the antivirus software running has gained you nothing. And probably cost you in performance degradation, reliability and endless false positives, even if not in monetary cost if it's a free one.
So, feel free to go without antivirus. Not because you're safe without it, but because you're not safe even with it. All it is doing is making you feel good that you're doing something about viruses.
Antivirus sucks in 2009.