Generally what that EULa is warning you about is that you can potentially enter settings that are
downright silly (and cannot/willnot ever do anything useful) but your monitor may still try to actually show an image in the first place.
These modes are potentially dangerous in that they can be over-driving your monitor inputs or graphics card outputs as they can force the RAMDAC to operate very close to their maximum rate for long periods (i.e. high resolution with super high refresh rate) which could cause the component to overheat. The same goes for whatever is on the receiving end of this signal, if the monitor is "dumb" and simply tries to show the image rather than doing a sanity check and simply dropping the signal on the floor like a sensible monitor would (along with displaying a "Signal out of range" OSD image) it is possible that the signal voltages or extreme fast voltage swings may cause similar component overheating and damage.
As Moab states the software will perform a check to make sure that the graphics card can put out the signal, but as far as that EULA states you have the possibility (however unlikely) that you can cause damage and they are having no part of it. As far as they are concerned there are some basic standard resolutions and frequencies that they have to support, and most monitors are friendly enough to let the graphics card know what it is expecting.
Past that as far as they are concerned while the chances are that you know enough not to send "bad" signals to your monitors this is their "here be dragons, you're on your own from now on" page.
-=EDIT: Comment Cleanup =-
One thing that would be useful to do would be to Google the make/model of your monitor and find out its "Horizontal refresh rate" as typically this is orders of magnitude higher than the vertical refresh rate (I think 50kHz is not uncommon) and make certain that whatever resolutions/refresh rates you end up with fit in your monitors horizontal and vertical refresh rates. Other than that the actual resolution should not matter too much, you just need to make sure that you are within the specifications of your monitor.
As an example my monitor shows that it can accept a vertical refresh rate of 55 to 75Hz and a horizontal rate of 24 to 80kHz, though I'm not sure how this applies to digital HDMI signals and I expect this is where the Bandwidth figure comes into it at 148MHz, the signal data rate on the Nvidia custom resolution page should not exceed this "pixel data rate" or damage could potentially occur. Basically make sure that none of the timing signals the card is going to output is outside of what your monitor expects.
Also, in the Custom Resolution page int he Nvidia control panel if I leave the "timing" set at "Automatic" then it seems to give settings that match my monitors settings with a horizontal rate of 67.5kHz, vertical rate of 60Hz, and a pixel clock of 148.5MHz. These seem to correspond quite well to the data-sheet for my monitor so I would look at what timing rates are shown for you and never exceed those rates. Other than that you should be fine to go a bit on the low side, but I don't know what a particularly low rate might do... YMMV, there may be dragons here, no responsibility, yadda yadda :)