Remote desktop needs a high quality, low packet loss connection rather than high bandwidth. If you are using WiFi it is entirely possible to have high bandwidth, but with interference causing a high level of packet retransmissions.
I've been using Remote Desktop on <10Mbps connections without the delays you are experiencing, the difference is that the connections are all wired, low interference and while the bandwidth might be low it is reliable.
In simple terms, in order to display a screen image Remote Desktop has to break down the screen you are viewing into small blocks and send them to your second computer.
If, during transmission, one of those blocks is corrupted then the program cannot display an image because you would have either a black block or a block showing wrong (old) image data. That might be fine briefly for a small number of bad blocks, but after a certain point all you will see is noise. The data may be delayed but the RDC program has to let the network stack do it's job and deal with retransmissions so it holds up the frame while waiting for that block to appear. This causes a delay.
Over Wi-Fi you can have interference from a huge number of devices, particularly in the 2.4GHz band. Microwaves, cordless phones, other WiFi networks, Bluetooth headphones, remote control toys and lots of other things can all use that band of frequencies and cause interference. Your router may well be the strongest signal and you get high bandwidth, but with a lot of background noise you can still get enough retransmissions that make a connection "poor" by comparison.
It is even possible that the WiFi device in the viewing machine has a poor or loose antenna or something there is making it difficult to send data back to your router to the Remote Desktop machine.
What might be fine for browsing the Internet, where delays due to retransmissions can be hidden by other parts of the page loading, is not necessarily fine for something where you need all the data to be there reliably at the right time.
Ethernet should be more reliable. If it definitely more reliable then interference is your problem. You might need to find a less congested WiFi channel, to switch to 5GHz if possible, or use an ethernet cable connected to a more local wireless access point in order to reduce the chances of interference.