I believe that there are several problems regarding custom firmware for commercial access points/ routers.
Firstly, If you decide to make a custom operations system/ firmware for a router, you will have to fit it into the flash memory of the desired router(2,4,8,16MB) and that means a lot of customisation to the code and optmization will have to be performed. However you can install an embedded shell like blackbox and you will have a lot of the standart functionality for next to nothing.
Secondly, the software that you choose to use withing the firmware will have to work with very limited resources(RAM,CPU).
Thirldy, I do not think that you will be able to obtain the precise hardware specifications of the platform, which in theory means that you may come to the situation in which you need to write a patch for an unknown problem.
All of the above make the firmware quite different from a typical operating system(BSD,Gentoo).
If you do decide to go with an available solution like DD-WRT or similar, do note that you need to have a device with compatible hardware(Broadcom, Atheros).