Make sure your local agent is ready
Just because you can ssh into your bastion server without specifying your key path or getting prompted for the password, doesn't mean that your ssh agent is running and holding your key. Some modern OSes (eg: OSX) handle this for you.
On your local machine
$ ssh-add -L
ssh-rsa ObahfCbvagGbLbhSbeHfvatEBG13== ~/.ssh/mykey.pem
ssh-rsa LbhNerWhfgJnnlGbbPyrireEBG13== ~/.ssh/sharedkey.pem
That means your agent is running and has your key.
$ ssh-add -L
The agent has no identities.
That means you haven't added any keys to your agent. Fix that with:
ssh-add ~/.ssh/mykey.pem ~/.ssh/sharedkey.pem
Make sure your remote agent is ready
SSH into your bastion server and repeat the check from fig.1 & fig.2. However, the error you are more likely to get is this:
$ ssh-add -L
Could not open a connection to your authentication agent.
That most likely means that your SSH client is not forwarding your authentication agent connection.
You can force this with the
-A flag (as long as the sshd config on the server allows it, which is the default).
$ ssh -A bastion.ec2
Make sure you're using the right keys
If you have added keys to your agent, your agent is forwarding, and your remote agent lists your local keys. There is only two likely reasons you are not getting a connection. Either you are not using the right key or you are not using the right username.
Output the public counterpart to your private key:
$ cd .ssh
$ ssh-keygen -y -f mykey.pem
$ ssh-keygen -y -f sharedkey.pem
These should be the same as what you were seeing from
ssh-add -L up to the
== near the end.
Now one way or another you have to get into the box you are failing to connect to and look at the content of the
$HOME/.ssh/authorized_keys file for the user you are trying to connect as. You need make sure the public key you output with the command above is in that file on a line by itself. You can't trust that the
sharedkey.pub that the Bro 2 cubes emailed you is right. Verify! This may require getting someone else who can SSH in as that user to get you the authorized_keys file or getting root access. If you've come this far and it still isn't working, you are beyond taking shortcuts.
Make it easy
Hopefully the steps above have gotten you in. Now let's make this headache go away for as long as you are using this workstation.
Configure your local ssh-client
# A lot of people put an IdentityFile line in this Host * section.
# Don't do that unless you will use only 1 key everywhere forever.
# You want to make sure you always forward your agent to this host.
# But don't forward to untrusted hosts. So don't put it in Host *
# Go a head and put the IP here in case DNS ever fails you.
# Comment it out if you want. Having it recorded is a good backup.
# You don't want to create a proxy loop later, so be explicit here.
# SSH should try using all keys in your .ssh folder, but if you
# know you want this key, being explicit speeds authentication.
# Connect effortlessly by hostname or IP address
# This assumes that your internal DNS uses the fake TLD ec2
# This assumes that 172.31.0.0 is your C-Class subnet
Host *.ec2 172.31.*
# This command says proxy all ssh connections through bastion as if
# you had done an ssh -A
ProxyCommand ssh -W %h:%p bastion.ec2
# These next lines are documentation you leave as a love letter to
# your future self when all else fails or you have to help a
# coworker and decide to look at your own config.
# ssh-add ~/.ssh/*.pem
# ssh -At bastion.ecs ssh firstname.lastname@example.org
If you take nothing else away from fig.7 it should be the proper use of
Auto populate your .bash_profile
You don't want to have to do
ssh-add manually every time you log into your machine.
~/.bash_profile is a script that runs each time you login**. Put the line from fig. 3 in there and you should always have your agent ready.
** Don't confuse this with
.bashrc which runs for each new [interactive] terminal. Your agent continues running even if you close all your terminal sessions. No Need to reload your keys
Alternative to using .bash_profile
I have also created a gist that add an OSX/macOS Launch Agent. You can use that method for starting your
ssh-agent on boot. It is very easy to install:
curl -sSL https://gist.github.com/RichardBronosky/429a8fff2687a16959294bcee336dd2a/raw/install.sh | bash