Browser caches are made up of files and metadata (used for chronology, address autocompletions and other things).
As you correctly guessed, cache deletion is (for the files) nothing more than file deletion.
It is actually something less, in that:
some implementations (I'm pretty sure about Internet Explorer 7, and therefore I suspect IE8 onwards) do not actually delete the metadata but just mark the archival space as reusable. Inside the .DAT file, the metadata is still there, so I could try and extract where and when have you been even if I can no longer have a copy of the actual file contents, but only its address (of course if the file hasn't changed, I can re-download it). This was a big issue with IE5 and IE6, then got somewhat ameliorated, but I don't know how much.
some browser actually rely on the cache index to know what files are stored in the cache and where. It may happen that some files drop "off" the index, and remain orphaned in the cache directories. That's how you can purge your browser cache, run a tool such as Piriform's CCleaner, and have it tell you that there is still space occupied by the browser cache. CCleaner just walks the cache folder hierarchy and reports whatever it finds, without caring if it is not listed in the browser cache index, thereby reaping also "orphan" files.
But these above are limited and possibly even no longer existing issues. As far as disk files are involved, cache files are still recoverable even if "deleted", until the physical space they occupy actually gets overwritten by new information.
To "securely" delete your browser cache you need to either clear it from the browser and then clear the free space of the disk (using Defraggler, SDelete, Eraser or such utilities), or you need to not clear it from the browser and delete it from another utility (e.g. the already mentioned CCleaner in "Secure Delete" mode).
Even so, files that have already been churned by the cache, i.e., created and deleted by the browser, will remain in "unsafely deleted" state. Having already been deleted, they won't be seen by the secure-delete utilities, and need to be sanitized with a disk free space sanitization which, on large disks, can be quite slow. One workaround is to set up a very large cache with infinite retention time, so that no file will ever be deleted by the browser (supersede pragmas, though, might come as a nasty surprise).