This is confusing since I know that Server 2003 allows thousands and thousands of entries in a single ipsec rule and I foolishly assumed 2008 and above also did.
I had received the same error and thought it was due to an overlap of adding a subnet that I had already added a single ip from the subnet and I spent a great deal of time going through the list until I found the overlapping ip, but when I continued to get the error on future ips and subnets I realized I needed to dig some more.
Years ago I had research as much as I could and found a number of articles that stated 2003 did not have a limit. In fact I have one production server that has over 20000 entries in a rule. (that is not a typo - it took years to get to tht number adding on average hundreds per month)
I am glad I found this article as it will save me a lot of time. I took the suggestion here and created a new rule and subnets that returned an error previously are fine in a new rule, so superuser answers a question for me once again.
It's too bad there is not an easy way to export and re-import single rules, the snapin appears to only allow all or none.
Probably a way to do it with powershell, but I guess that should be another question thread. If there happens to be a thread that covers that here, would appreciate the search string to look for or a link.
Edited to add: My OS is Server 2008 Standard x64 SP2