Sometimes, rules and other similar settings get goofed up. Try opening Outlook with this from the Run box:
outlook /cleanrules. Then try if it is not already gone.
This may also be relevant:
I will keep the rest of the original answer as it may be helpful to someone down the road.
By their very nature, delegate information is stored on the Exchange server, otherwise, if you were not connected, how would it know who to authorize in your absence?
So you cannot delete them until you are connected, and your Outlook can update the removal on the Exchange Server.