I'm only going to address question (1).
Your problem is KEYTIMEOUT. I quote from zshzle(1):
When ZLE is reading a command from the terminal, it may read a
sequence that is bound to some command and is also a prefix of a
longer bound string. In this case ZLE will wait a certain time to see
if more characters are typed, and if not (or they don't match any
longer string) it will execute the binding. This timeout is defined
by the KEYTIMEOUT parameter; its default is 0.4 sec. There is no
timeout if the prefix string is not itself bound to a command.
That 0.4s is the delay you're experiencing after hitting ESC. The fix is to set KEYTIMEOUT right down to 0.01s in one of the shell startup files:
export KEYTIMEOUT=1
Unfortunately this has a knock-on effect: Other things start going wrong…
Firstly, there is now a problem in vi command mode: Typing ESC causes the cursor to hang, and then whichever character you type next gets swallowed. This is because ESC is not bound to anything by default in vi command mode, yet there are multi-character widgets that start with ESC (cursor keys!). So when you hit ESC, ZLE waits for the next character… and then consumes it.
The fix is to bind ESC to something in command mode, thus ensuring that the something gets passed to ZLE after $KEYTIMEOUT centiseconds. Now we can keep bindings starting with ESC in command mode without these ill effects. I bind ESC to the bell character, which I find to be even less intrusive than self-insert (and my shell is silenced):
bindkey -sM vicmd '^[' '^G'
Update 2017:
I have since found an even better solution for binding ESC — the undefined-key
widget. I’m not sure whether this widget was available in zsh when I originally wrote this answer.
bindkey -M vicmd '^[' undefined-key
Next problem: There are by default some two-key widgets starting in ^X in vi insert mode; these become unusable if $KEYTIMEOUT is set all the way down. What I do is unbind ^X in vi insert mode (it's self-insert by default); this allows those two-key widgets to continue working.
bindkey -rM viins '^X'
You lose the binding for self-insert, but you can bind it to something else of course. (I don't, since I have no use for it.)
The last problem (I've found so far): There are some remaining default keybindings that we "lose" due to setting $KEYTIMEOUT right down, to wit: those starting with ESC in vi insert mode which are not cursor keys. I personally rebind them to start with ^X instead:
bindkey -M viins '^X,' _history-complete-newer \
'^X/' _history-complete-older \
'^X`' _bash_complete-word
Update 2018:
It turns out the entire section above (after “Update 2017”) is not necessarily required. It’s possible to set the META key to be equivalent to ESC in keyboard mappings using:
bindkey -mv
It is therefore possible not to unbind ^X, and to access the keybindings that start in ESC by pressing META as a leader instead (ALT or OPT on modern keyboards).
If you have access to the book From Bash to Z Shell by Kiddle et al., the equivalence of ESC and META in keybindings is discussed in the Chapter 4 sidebar on pp. 78–79.
i
twice to get back to insert mode, I would highly recommend this fix!