You can strip any linux distro down to the bare bones - the simplest thing to do is take a distro you already are familiar with and boot into run level 1 - the standard 'single user mode'. You won't have networking or any daemons, but you may have a desktop, depending on what you're using. You could define a custom run level to do whatever. Run level 4 is usually used for this. It's undefined, so you can make it whatever you want.
Alternatively, take a look at the linux distros tuned for smallness
Linux is not exactly famous for speedy boot times, but the less you run at startup, the faster it will be.