Xmodmap cannot do what you're asking for, but it can bring you close.
The basic syntax for modifying the behavior of a key is
xmodmap -e "keycode KEYCODE = KEYSYMBOL1 KEYSYMBOL1 KEYSYMBOL3 KEYSYMBOL4"
xmodmap -e "keysym KEYSYMBOL = KEYSYMBOL1 KEYSYMBOL1 KEYSYMBOL3 KEYSYMBOL4"
KEYSYMBOL2 gets triggered when pressing the key while pressing Shift.
KEYSYMBOL3 gets triggered when pressing the key while pressing ModeSwitch.
KEYSYMBOL4 gets triggered when pressing the key while pressing Shift + ModeSwitch.
KEYSYMBOL1 gets triggered when pressing the key with none of the above.
- ModeSwitch is AltGr by default (only works on some keyboard layouts).
We can't set Ctrl or Alt to be the ModeSwitch key, since no key can serve as two different modifiers.
However, we can turn CapsLock into the ModeSwitch key:
xmodmap -e "keysym Caps_Lock = Mode_switch Mode_switch"
To remap CapsLock + Ctrl to the CapsLock key, execute the following commands.
xmodmap -e "keysym Control_L = Control_L Control_L Caps_Lock"
xmodmap -e "keysym Control_R = Control_R Control_R Caps_Lock"
Note the CapsLock is the modifier key here, so you have to press this key first.
We can't remap CapsLock + Alt or CapsLock + Super to Escape in the same manner, since Alt and Super serve different functions when pressed alone and when used as modifier keys.
Every other key will work fine. Examples:
For CapsLock + ` (standard US layout), use
xmodmap -e "keysym grave = grave asciitilde Escape"
For CapsLock + Space, use
xmodmap -e "keysym space = KP_Space KP_Space Escape"
To make the remappings permanent, create/edit the file
.gnomerc in your home folder and save the corresponding commands in it.
To undo all remappings, simply execute