Assuming the modem is a PPPoE device, it would probably actually work. The switch switches around Ethernet (Layer 2) frames, and the router uses PPPoE to communicate with the modem. PPPoE works via Ethernet (as the name suggests), and seems to use its own discovery protocol to find the MAC address of the modem. It should be able to coexist with IP in one network. So, in theory, if you connected the router's "modem" ethernet port to the switch, and connected the router's "network" ethernet port to the switch using a second cable, it should work. Note that you will have to connect both the "network" ethernet port and the "modem" ethernet port to the switch. Test before installing, obviously. Note that you may have to fiddle around with crossover/non-crossover cables.
It doesn't make much sense, however, and can lead to various problems and security issues (do you want any PC in the network to be able to establish PPPoE sessions?). Instead of running the "modem" cable from the router to the switch and from the switch to the modem, just run it directly from the router to the modem. If you are working with wall jacks, you can connect two of them directly with a cable instead of using one cable to connect jack A to a switch and a second one connect the switch to jack B.