I need to tune my windows 8.1 for a web scraper

I am using thousands of threads for concurrent crawling

For crawling process, i am using my own written c# HttpWebRequest using crawler

However i am having problems when i use proxies

I dont know what causes error however established connection count keep stacking while nothing on the system can connect anything until i terminate the application, disable network card and enable back again

I am properly disposing every connection even if the connection get terminate unexpectedly

These are the tunings i have found so far but i wonder are there any other tunings i can make?

My computer is very strong so i have no hardware limitation

regedit tweaks TCPIP

  • set TcpTimedWaitDelay to 30 seconds
  • TcpFinWait2Delay : 30 seconds
  • seconds set MaxUserPort to 32k
  • EnableDynamicBacklog : 1
  • MinimumDynamicBacklog : 20
  • MaximumDynamicBacklog : 10k
  • DynamicBacklogGrowthDelta : 100
  • EnableTCPA : 1
  • TcpAckFrequency : 1
  • TCPNoDelay : 1
  • TcpDelAckTicks : 1
  • LocalPriority=4
  • HostsPriority=5
  • DnsPriority=6
  • NetbtPriority=7

cmd commands:

  • netsh int tcp set global autotuninglevel=disabled
  • netsh int tcp set global ecncapability=enabled
  • netsh int tcp set global rss=enabled
  • netsh int tcp set global chimney=enabled
  • netsh int tcp set heuristics disabled
  • netsh int tcp set global dca=enabled

my current settings

enter image description here

my computer is core i7 2600k @ 4.5 ghz = 8 threads 32 gb ram raid 0 ssd disks as C drive

After a while established connection counts never get decreased and keep increasing. When this happens, internet connectivity is lost

enter image description here

  • 1
    Using thousands of connections to scrape a site is bordering on abusive, any self respecting site owner would be tempted to find a way to block such behaviour. That said, the fact that you have to disable and reenable the network card is telling. Have you tried a different card? Have you looked for updated drivers? – Mokubai Mar 21 '17 at 6:38
  • @Mokubai the issue is not site related. That is another issue as you have mentioned (i mean having thousands of connections). I did not try different network card because i dont have. Yes drivers are all updated. Some weird bug happens in system and established connection count keep starting to stack. any ideas? – MonsterMMORPG Mar 21 '17 at 14:51
  • 1
    Despite what you've said in your question I'm tempted to say that you are not correctly disposing of the TCP and they are lingering around long after they should have been closed. Are you simply destroying pointers to the connection in your program or are you fully closing them down? If this is truly a problem in your code then it would be better to post an example of what you are doing to Stack Overflow. – Mokubai Mar 21 '17 at 15:03
  • If ESTABLISHED sockets are continually increasing, then you are not closing them out. TCP sockets, when no longer needed, should be closed using a specific process (see here: tcpipguide.com/free/t_TCPConnectionTermination-2.htm). There are TCP setting that might help if they linger in states like TIME_WAIT, but initially moving from ESTABLISHED to FIN_WAIT is all on the application, not TCP. – MaQleod Mar 21 '17 at 15:39
  • @MaQleod that is the issue. i am following every close handling rules. check here : codereview.stackexchange.com/questions/158359/… . it is not like starting to stack after application starts. application runs perfectly fine as expected for a while. lets say sometimes 30 40 minutes with hundreds of thousands of requests. all handled properly. then somehow some error bug happens. internet connectivity get lost and established count starts to stack – MonsterMMORPG Mar 21 '17 at 18:44

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.