The small switch is probably the best solution, but there is another option that you may come across that I'll just explain. You may find so-called Ethernet port doublers, which look like a twin phone adapter only Ethernet (RJ45) sized. These can work - but will only work at 10 or 100Mb Ethernet not Gigabit, and you have to have two doublers - one at your wall port and another at the patch-panel with cables going to two real switch ports.
The reason they work is that there are four pairs or cables in Ethernet cable, but only two pairs are used for 10/100. The splitter is wired so that the extra two unused pairs are connected to the usual pins on the 2nd port. Gigabit Ethernet uses all four pairs, so if you use a port doubler - you max out at 100Mb.