Here's what I think happened:
Capacitors lost their capacitance somewhere in computer. I believe that it's most likely in the power supply. Here's why I think so: (I assume that computer has ATX power supply because you mentioned orange LED and the switch on the power supply, I don't know if the booting process is same for AT power supplies) Once you press the power button, signal goes from motherboard to power supply to start it. Once power supply starts providing power, motherboard waits for for power supply to stabilize. It should normally take less than a second for it to stabilize, but if the capacitors are bad, it may take several on/off cycles for them to fully charge and start providing stable enough power for motherboard to start booting the computer.
It could be that the bad capacitors are on the motherboard itself, but I think that it's a fault related to PSU.
Unfortunately, it's not going to be easy to repair the issue in an economically justifiable manner. Some options are to replace motherboard or PSU depending on which is broken (you'd need a working PSU to determine that). The problem with that solution is that motherboards of that time are very rave now, so it may be difficult to find compatible motherboard. Other option is related to power supply. If it is ATX power supply, as I suspect it to be, you are either going to need to find old ATX power supply using ATX 1.x standard or buy a high end new power supply and hope that it can provide power to old computer. The problem is that in old ATX power supplies, 3.3V and 5V lines were used to supply most of the power to the system. In new ATX 2.x power supplies, 12V lines are used to provide most of the power to the system.
Another option which may be better that first to is to replace the capacitors, if they are the problem. The problem with this option is that you need to find experienced technician who will replace the capacitors. New capacitors themselves probably won't be very expensive, but technician may charge a lot for his services.
To check if I'm right about capacitors, open the computer case and take a look at them. If they look like something from this article, it's probably them.