It's unlikely that this has anything to do with latex.
There are no single-character codes for arrow keys; instead, they send control sequences. For example, the up-arrow key typically sends
ESC [ A.
terminfo (more likely the latter these days) to interpret those sequences, based on what kind of terminal it thinks you have, specified by the
$TERM environment variable. You can see these sequences by, for example, typing
control-V followed up an up-arrow at a shell prompt (
control-V says to interpret the next character literally).
vi doesn't have enough information to know what
ESC [ A means.
There's also an alternate set of codes; for example, up-arrow might be sending
ESC O A. In
xterm, this mode can be toggled via control-middle-click and enabling or disabling "Application Cursor Keys".
Note also that you can use
l for left, down, up, and right, respectively. This can be easier to use than the arrow keys once you get used to it; you don't have to move your right hand off the home row of the keyboard.
What is the output of
echo $TERM, typed at the shell prompt? What terminal emulator are you using (
xterm, etc.)? What version of
vi are you using (
vim, something else?) What OS are you on? Are you using Cygwin?