How can I configure the Modem to get maximum download speed for Bit torrents?
You need to configure port forwarding , depending on the modem/ router the steps will vary. Find our what port your client is listening on then visit http://portforward.com/ for steps to configure the modem / router
Not exactly sysadmin related, but what the hey:
What a huge can of worms.
Many torrent clients (like Azureus) support UPnP so if your router also supports UPnP (and it is enabled), then the client can take care of it for you.
Other than that, you need to find out what port your software uses (it's usually a high port, like 16000 or something) and forward that port in your router to the downloading machine. Most software will allow you to choose your own port, so it shouldn't be too hard to find out what it is.