How can I make my ISO_Level5_Shift arrow keys work in Java Swing GUIs?

Question

I have a keyboard layout that uses ISO_Level5_Shift to provide arrow keys. From my layout symbols file:

key <AC06> { type[Group1]="EIGHT_LEVEL", [ d, D, ampersand,  U2227, Home,  Home,  Greek_delta,   Greek_DELTA   ]};
key <AC07> { type[Group1]="EIGHT_LEVEL", [ h, H, parenright, U27E9, Left,  Left,  Greek_eta,     Greek_ETA,  U210F  ]};
key <AC08> { type[Group1]="EIGHT_LEVEL", [ t, T, parenleft,  U27E8, Down,  Down,  Greek_tau,     Greek_TAU     ]};
key <AC09> { type[Group1]="EIGHT_LEVEL", [ n, N, slash,      U2115, Right, Right, Greek_nu,      Greek_NU      ]};
key <AC10> { type[Group1]="EIGHT_LEVEL", [ s, S, underscore, U2237, End,   End,   Greek_sigma,   Greek_SIGMA   ]};

These work in most programs (Firefox, Eclipse, Vim, ...). Unfortunately they do not work in any Java Swing GUI that I have ever used. In particular, they do not work in IntelliJ IDEA, and this is what has been bugging me in particular.

Is there something I could change in my layout, or Java-related environment variables, or IDEA configuration, that might fix this problem?

Answer 1

OK I found a solution. It is not really ideal, but it does get the desired behavior.

First, I dumped the complete state of my keyboard configuration using

$ xkbcomp $DISPLAY - > now.xkb

Then I found the lines

interpret Overlay1_Enable+AnyOfOrNone(all) {
    action= LockControls(controls=Overlay1);
};

and changed it to

interpret Overlay1_Enable+AnyOfOrNone(all) {
    action= SetControls(controls=Overlay1);
};

which prevents the modifier from being "sticky" ie it only applies while you are holding the key down.

Then I took the key that used to be my ISO_Level5_Shift:

key  <TAB> {
    type= "ONE_LEVEL",
    symbols[Group1]= [ ISO_Level5_Shift ]
};

and changed it to Overlay1_Enable:

key  <TAB> {
    type= "ONE_LEVEL",
    symbols[Group1]= [ Overlay1_Enable ]
};

Then for every key where I wanted the change to take effect, I added an overlay definition:

key <AD07> {
    type= "EIGHT_LEVEL",
    overlay1= <PGUP>,
    symbols[Group1]= [               g,               G,        asterisk,               G,           Prior,               G,     Greek_gamma,     Greek_GAMMA ]
};

Then a re-applied the whole thing with

$ xkbcomp now.xkb $DISPLAY

---

Useful documentation:

• XKB Page on ArchLinux Wiki
• Doug Palmer's XKB Guide

Answer 2

I was having issue with this as well, here's my solution using caps lock key as the overlay switch to enable emacs/vim like navigation, hopefully it will help anyone looking to do something similar.

// Emacs like keys with CAPS as overlay switch.

// Using ISO_Level3_Shift or ISO_Level5_Shift would also make
// most applications work and would have been more flexible,
// however they don't work in Java Swing apps (e.g. IntelliJ IDEA)
// but using overlay works

// To enable, save this file as /usr/share/X11/xkb/symbols/emacs and run:
//
//   setxkbmap -option '' "emacs"
//
// However it may not persist and can get reset back to the default by other things.
// Alternatively, insert the following into /usr/share/X11/xkb/rules/evdev.xml
// ...
// <layoutList>
//   ...
//   <layout>
//     <configItem>
//       <name>emacs</name>
//       <shortDescription>en</shortDescription>
//       <description>English (US, Emacs)</description>
//       <languageList>
//         <iso639Id>eng</iso639Id>
//       </languageList>
//     </configItem>
//   </layout>
//   ...
// </layoutList>
// ...
// Then you should be able to choose 'English (US, Emacs)' in a keyboard preference
// GUI such as fcitx and have it persist.

default partial alphanumeric_keys
xkb_symbols "basic" {

    // Base keyboard functionality, using 'us' here
    // See definitions in /usr/share/X11/xkb/symbols
    // e.g. 'include "us(intl)"' where 'intl' is defined inside the 'us' file
    include "us"

    // Press Shift_L + Shift_R together as an alternative way to toggle Caps_Lock,
    // then turn CAPS key to enable overlay1 mode when and only when holding down.
    include "shift(both_capslock)"
    key <CAPS> {actions[Group1]=[SetControls(controls=overlay1)]};

    // Emacs like navigation keys when holding down <CAPS>
    // e.g. caps + n to go down
    key <AB05> {overlay1=<LEFT>}; // b
    key <AC04> {overlay1=<RGHT>}; // f
    key <AD10> {overlay1=<UP>  }; // p
    key <AB06> {overlay1=<DOWN>}; // n
    key <AC01> {overlay1=<HOME>}; // a
    key <AD03> {overlay1=<END> }; // e
    key <AC05> {overlay1=<ESC> }; // g

    // Emacs like editing keys when holding down <CAPS>
    // Redo/Undo only work with applications that understand the them
    key <AB10> {overlay1=<UNDO>}; // /
    key <UNDO> {[Undo, Redo]};    // Shift + / -> Redo
    key <AC03> {overlay1=<DELE>}; // d

    // VIM like navigation keys when holding down <CAPS>
    key <AC06> {overlay1=<LEFT>}; // h
    key <AC09> {overlay1=<RGHT>}; // l
    key <AC08> {overlay1=<UP>  }; // k
    key <AC07> {overlay1=<DOWN>}; // j

};

Answer 3

In particular, they do not work in IntelliJ IDEA, and this is what has been bugging me in particular.

There is the workaround − to map desired keys on IntelliJ IDEA side.

1. Go to Settings ⇒ Keymap.
2. Search for a key which doesn't work when the “iso level shift" key combination pressed (say “Up” key).
3. Choose “Add Keyboard Shortcut”.
4. Press the “iso level shift” key combination which doesn't work as “Up” key in IntelliJ.
5. Apply.

Voilà, the “iso level shift” key combination behaves as “Up” key even in IntelliJ IDEA.

Answer 4

I have found another solution, that is slightly more work, but ultimately works more robustly across more different software.

The basic idea is to skip XKB and tap directly into the kernel event handling mechanism.

Once you do that, you can control precisely what key codes all software sees.

I ended up creating a generic key remapping tool to accomplish this.