I am running an online game for kids but lately I have been experiencing attacks from some vile individuals who just like ruining it. I assume (based on Windows log) that they connect hundreds time with resending the packets so the server runs out of tcp connection slots. The server (software for that game) disconnects these unassigned connections after like 5 seconds, but recieving thousands of messages, it gets jammed. My only idea is - could a proxy (HW or SW) be used for filtering incoming traffic from particular IP? So these data would never reach the server. Thanks
You could use a reverse proxy to achieve what you want, provided you are running a web based application, however this just shunts the effort onto the proxy, and it isn't the ideal solution for your problem.
This type of problem is usually solved by using a firewall. All firewalls can filter IP addresses, or ranges of IP addresses, and if you are running an online service such as you describe, it is definitely something that should be in the mix.
Additionally, a modern full featured firewall can also automatically block IP addresses with excessive open connections, to a threshold you define. This eliminates the need to continually and manually "chase" malicious activity that may change IP addresses to work around any hardcoded blocks.