Wow. I had never noticed that, but x-popup-dialog seems to be a problem for me too. At least the example dialog crashed for me. How do you get customize to bring up a dialog? I can't reproduce it.
In general the rule is: if you do it from the keyboard it won't use up a dialog. e.g. use C-x k instead of the mouse to close a buffer and you'll get no dialog. Another, more heavy-handed way, is to set use-dialog-box to nil (though that doesn't stop x-popup-dialog from working if called directly so it may not fix the problem). To fix that you would probably have to advise x-popup-dialog, reimplement it using for example completing-read and never call ad-do-it. Or you could fix the bug. I'm not sure which would be easier :-)
moreinfo: debbugs.gnu.org/cgi/bugreport.cgi?bug=2877 Maybe someone could get the ball rolling again?