Sounds perhaps not quite but a little bit like what would happen if forgetting to plug the 2x2 cable into the motherboard. I think I had it with the P4.. an old system.. probably your system is more modern and perhaps only the motherboards for the p4, that needed it, but anyhow, if your motherboard has that socket, then plug it in..
I think in the p4 case perhaps it always did the few seconds thing. didn't need to unplug it and plug it back in for it. so perhaps not quite the same symptoms.
Also take the motherboard out of the case and try
And remove screws from it.. (I did once have a weird MBRD that didn't power up at all 'cos one screw in a hole designed for a screw, but not meant to take it.. shorted it. was strange. I don't think your case.. but worth trying to take the MBRD out of the case)
And do try another PSU. that's an obvious thing to try.