2

How to setup SSH key signing process every time some Ansible stage runs? Or at some interval at least.

I have pipeline which creates VM's with the Terraform and then runs Ansible. At cloud-init stage Host keys are created and signed as well public User CA key are configured. Client/Host CA authorities are configured/running in the HashiCorp Vault. So, at this point, no matter what VM's are provisioned, i can SSH into every of them as i have hosts CA public key @cert-authority *.example.com ecdsa-sha2-nistp521 AAAAB3NzaC1yc2EAAA... set globally in /etc/ssh/ssh_known_hosts. All i need to do is to create new personal key and to sign it because my TTL is quite short.

But this does not work in Jenkins. Jenkins stores all SSH keys under /var/lib/jenkins/.ssh and by default there is nothing. If just for the testing reasons i do copy my personal user (or generated) keys and certificate + ssh config file into /var/lib/jenkins/.ssh, then Jenkins can run Ansible happily. But, i can't generate, sign and copy the jenkins keys every time i do some commit in my infra repo. Creating long living certs also does not smell good.

What is the idiomatic automated workflow for SSH key signing and rotation?

1 Answer 1

1

I will list the "BAD" solution by myself. Bad, because key rotation there is mannual and TTL most likely will be long because we are lazy.

In the Vault you need to create a new "jenkins" SSH role with the TTL like 52w

Then, ideally on the Jenkins host itself you need to create new keys.

ssh-keygen -t ecdsa -b 521 -f /var/lib/jenkins/.ssh/id_ecdsa -C "jenkins@jenkins-01"

Then singn it by Vault

vault write -field=signed_key ssh-client-signer/sign/jenkinsrole public_key=/var/lib/jenkins/.ssh/id_ecdsa.pub > /var/lib/jenkins/.ssh/id_ecdsa-cert.pub

This means that Vault should be on the system. ... it's a dilemma - do you want to copy ssh keys (including private) over the network, which smells a bit, or you want to install Vault on the Jenkins machine. It's up to you.

After you have id_ecdsa-cert.pub certificate you can inspect it with ssh-keygen -Lf /var/lib/jenkins/.ssh/id_ecdsa-cert.pub and see is the Validity period, principals, etc. right.

Because most likely there will be drifts in the infrastructure, storing SSH config statically in the /var/lib/jenkins/.ssh/config will be painful.

Instead you can create ansible-ssh.cfg file with the content like

Host bastion
  HostName bastion.example.com
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_ecdsa
  CertificateFile ~/.ssh/id_ecdsa-cert.pub
  Port 22
  Protocol 2
  User ansible
  LogLevel INFO

This file can be (should be) located alongside with the rest of Ansible code.

Then, in the ansible.cfg file include SSH stanza

[ssh_connection]
ssh_args = -F ./ansible-ssh.cfg

This way Ansible will pick up always up to date SSH configs.

If you use Terraform then it is possible dynamically update SSH config file to include new hosts and what not.

This solution is not tied to the Vault only. It can be done quite easily with the ssh-keygen itself.

UPDATE: Eventually i ended up with small shell script which does authorization by using approle and then signs the cert. Systemd service and Timer unit executes the shell script which does the cert rotation every day. So at this point Jenkins uses it's own fresh User certs to run the Terraform and Ansible. https://stackoverflow.com/a/65875997/6651080

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .