I have AMD Athlon 3500+ (x86_64) CPU and I am okay with the Ubuntu package maintainer's kernel version (as of now it is 2.6.35-24-generic). What will I gain if I update to latest stable kernel (184.108.40.206)? A fraction of performance?
Only correct answer to this question is to actually go to http://kernel.org/ and check changelog for each stable version. In some versions, there could be performance improvements, in other bugfixes or there could be new device drivers added.
For example, here are logs for at this time current stable version: http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-220.127.116.11 Here's a more readable article: http://kernelnewbies.org/LinuxChanges
As a general advice, I'd say that it's best to use newest version provided by your distribution and let its developers worry about changes unless you need something from newest stable version from kernel.org
Also, there is danger in moving to newest version from kernel.org. For example one time in stable version, network driver used to delete firmware from certain Intel network controllers. It passed into stable because nobody happened to use affected controller in the development team. It was later discovered by early adopters. That is why distributions like to wait for a while before moving to the newest version.
It's mostly just bug fixes, so I doubt you'll see a lot of difference. One of the biggest changes was VFS Optimization, which would benefit you a lot...If you were using a multicore machine.
I'd just stick with 2.6.35 unless you're having issues.
There is an article dealing with the changes here if you don't want to wade through the kernel notes.