I did read the grammar section of xmodmap but still couldn't figure out how I could do it.
This isn't easily possible.
As the xmodmap man page states:
By default, AltGr is the Mode_switch key.
It is possible to map a different or additional (to AltGr) as the Mode_switch key, e.g. the right Ctrl key:
The problem however is that the key then won't work as normal Ctrl key anymore.
Btw: There is a question, which asks for precisely, how to use the Ctrl key as a modifier key: How to use Control as modifier with xmodmap?
and let me know if it's the result you're looking for.