Aside from viruses, the reason you're told to clear your browser's cache is because of the cookie files websites store locally. If an attacker had a virus on your computer already, it would be trivial for him or her to hijack your sessions that you didn't log out. They could just read your session ID from the cookies still stored on your computer, because you didn't log out nor delete them.
By simply logging out of a website, you mitigate this risk, as the particular session ID would be invalid. However, some credentials might be saved locally or through the session ID itself server-side (i.e. your username, previous form submissions).
As for deleting the browser's cachce to deal with drive-by-downloads, so long as you have an antivirus program that is running correctly, it should detect the virus as soon as it's downloaded onto the computer.
Assuming it was not detected, and the virus was written to disk, clearing the cache might further trigger the AV to inspect the files in the cache during their deletion. However, if your AV does not detect the file after it's downloaded into your browser cache, I would assume that the attacker has already executed the code on your machine.
In closing, clearing your cache won't really help you if the virus code has already been downloaded and executed on the target computer. However, there are all kinds of breadcrumbs present in a browser's cache (including temporary files and cookies). If an attacker was already present on the computer, or accessed the machine at a later date, he/she could use these things to possibly exploit your various online accounts, or analyze your browsing history (and use the results to improve man-in-the-middle attacks).
While it's probably overkill to clear your cache after every single session, the best practice would be to ensure you log out of websites when you're done using them, and periodically clear your browser's cache (twice a week should be sufficient, more if there is lots of disk space being used).
And as always, after visiting websites containing sensitive information, clear all temporary files left behind immediately, or browse using your browser's "private" mode (incognito).