Lets start with the obvious things and work our way down.
Physically different form factors of systems have different ways of interfacing with your motherboard - you can't get a desktop LGA or PGA processor into a smaller laptop socket with a different pinout. Phones and some laptops also use soldered on BGA processors. You'd be trying to fit a 1/4 inch peg into a 1/2 inch hole.
You'd also need 'motherboard' level support - with PCs this would be in the bios/uefi firmware to be able to bootstrap the processor. Intel's sandy bridge and ivy bridge processors shared a socket, and could be swapped between boards of either generation, but sandy bridge boards needed a firmware update. As such your firmware would need to be able to communicate with either.
Secondly, they run different Instruction Set Architectures . Assuming you had a magical motherboard with a chipset that could speak to both a specific ARM and Intel chip... none of your software would work without a compatibility layer. You could in theory outfit an OS to run software from another architecture, like rosetta, but simply swapping chips would result in breakage in software. And this is assuming a hypothetical magical system, that is compatible with two completely different chip designs!
In short, if you had two chips with ISA compatibility, identical pin outs and firmware support, you could swap them. If you tried to put an arm chip into a intel board, or a intel processor into a arm phone (which would be monumentally stupid considering phones have soldered on processors), things would very likely not work. Things might burn out since pinouts are different. It simply would be a disaster.
Finally, clock speeds of processors these days has less and less relevance to actual performance. There was a massive dip in processor clock speed post pentium IV on intel processors. Comparing clock speeds is useful within the same family and generation of processors taking into account cache and core count. Comparing an arm processor and a intel processor of identical speeds properly would mean throwing carefully designed sets of work at them to see how they perform.
One does not simply switch chips between ISAs or formfactors without a full engineering facility.