You could actually build a kernel to your specific hardware if you wanted to. I think a good distribution to illustrate this is gentoo - you can run kernels tuned to your hardware or run genkernel.
You could use a generic kernel, which is bigger, or build a kernel to your needs. Forget something that can't be loaded as a module, and you rebuild the whole kernel.
There's of course kernel modules (which is how many third party drivers are packaged), but the point of having a default sane set of kernel modules is to be able to know that your system is reasonably likely to work. Imagine needing to spend a few hours compiling your kernel because you forgot to enable X (this happened to me, the last time I tried to install gentoo)!
You do not, however have the drivers for all devices - you often end up installing drivers yourself - for example, nvidia has seperate drivers outside the noveau packages many distros have (and this is outside the kernel) and the same is true of many network cards. There's also mechanisms that just load necessary modules as needed
If you want a minimal set of drivers, on a slim kernel, built to the specifications of your own system, do it. Just remember its a lot of work, and the generic kernel is usually a sane set of tradeoffs the distro developers chose cause its reliable.