Using signed SSH certificate in AAP

AAP support using signed SSH certificate as machine credential. Here are example instructions setting it up.

Signing certificates and distrubte CA public key

In this example, we are going to use my SSL certificate. First, we need to extract the public key from the certificate:

$ openssl x509 -pubkey -in automate.nyc.crt -noout > x509-key.pub

Then, we need to convert it to OpenSSH public key

$ ssh-keygen -i -m pkcs8 -f x509-key.pub > automate.nyc.pub
$ ls -l
total 16
-rw-r--r--. 1 echong echong 1927 Feb 13 10:19 automate.nyc.crt
-rw-------. 1 echong echong 1675 Feb 13 10:19 automate.nyc.key
-rw-rw-r--. 1 echong echong  381 Feb 13 10:25 automate.nyc.pub
-rw-rw-r--. 1 echong echong  451 Feb 13 10:21 x509-key.pub
$ cat automate.nyc.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDj9HfV3SAlTWVdDYjts7nF0+16....

Next, we will generate another set of SSH keys and signed it with our certificate key:

$ ssh-keygen -C 'signed with automate.nyc.crt' -f my_ssh
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in my_ssh.
Your public key has been saved in my_ssh.pub.
The key fingerprint is:
SHA256:5IqjQ/lo5zCDaIjkEGsIg7nIYjTU+z72F27a8C0k91g signed with automate.nyc.crt
The key's randomart image is:
+---[RSA 3072]----+
| ..              |
|o. .             |
|*o  .   .        |
|*=..   o         |
|B= ..   S        |
|@.o  o o + E     |
|=++o+ ..= =      |
|. +=o=  +*..     |
| ..+o oo+o..     |
+----[SHA256]-----+

The key ID will be echong_ssh_automate_nyc and valid for 52 weeks for username echong only. To add multiple usernames/principals, use a comma separated list.

$ ssh-keygen -s automate.nyc.key -I 'echong_ssh_automate_nyc' -V +52w -n echong my_ssh
Signed user key my_ssh-cert.pub: id "echong_ssh_automate_nyc" serial 0 for echong valid from 2023-02-13T10:50:00 to 2024-02-12T10:51:05
$ ls -l
total 28
-rw-------. 1 echong echong 1927 Feb 13 10:19 automate.nyc.crt
-rw-------. 1 echong echong 1675 Feb 13 10:19 automate.nyc.key
-rw-rw-r--. 1 echong echong  381 Feb 13 10:27 automate.nyc.pub
-rw-------. 1 echong echong 2675 Feb 13 10:29 my_ssh
-rw-r--r--. 1 echong echong 1711 Feb 13 10:51 my_ssh-cert.pub
-rw-r--r--. 1 echong echong  582 Feb 13 10:29 my_ssh.pub
-rw-rw-r--. 1 echong echong  451 Feb 13 10:21 x509-key.pub
$ ssh-keygen -L -f my_ssh-cert.pub 
my_ssh-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:5IqjQ/lo5zCDaIjkEGsIg7nIYjTU+z72F27a8C0k91g
        Signing CA: RSA SHA256:Spn7YWfSWItTuSbzp+0zJKxBSTnYLpcRIcvtNrt9NBE (using rsa-sha2-512)
        Key ID: "echong_ssh_automate_nyc"
        Serial: 0
        Valid: from 2023-02-13T10:50:00 to 2024-02-12T10:51:05
        Principals: 
                echong
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Now we need to distribute the CA pub key to all the hosts will be accepting this SSH key. Copy the signing SSH public key string, in our case it will be the content of the converted automate.nyc.pub. On the target hosts, update the /etc/ssh/sshd_config with parameter TrustedUserCAKeys point to the public keys file:

TrustedUserCAKeys /etc/ssh/ca.pub

where the ca.pub file holds all the CA pub keys, one per line. Restart sshd to update the configuraiton. This will allow all users/principles listed in the signed SSH key to login the the host. The CA public key can also assined per user by entering into user’s ~/.ssh/authorized_keys file.

Setup AAP machine credential

On AAP machine credential page, paste the content my_ssh private key to the SSH Private Key and the content of my_ssh-cert.pub to the Signed SSH Certificate text boxes. If the my_ssh key is generated with a passphrase, enter the Private Key Passphrase text box too.

When using this credential to connect to the target host, the following will be logged in syslog:

Feb 13 11:26:32 aap-db1.lab.automate.nyc sshd[109844]: Accepted publickey for echong from 192.168.0.101 port 53348 ssh2: RSA-CERT SHA256:5IqjQ/lo5zCDaIjkEGsIg7nIYjTU+z72F27a8C0k91g ID echong_ssh_automate_nyc (serial 0) CA RSA SHA256:Spn7YWfSWItTuSbzp+0zJKxBSTnYLpcRIcvtNrt9NBE

Noticed the Key ID echong_ssh_automate_nyc we used to sign the SSH key, the thumbnails of the ssh key and signing key are logged in the message.

Summary

These are where the files go:

File	Description	Location
automate.nyc.pub	The public key generated from signing certificate	/etc/ssh/ca.pub or ~/.ssh/authorized_keys
my_ssh	My private key	Copy the content to AAP SSH Private Key text box
my_ssh-cert.pub	My signed public key	Copy the content to AAP Signed SSH Certificate text box
my_ssh.pub	My public key	No use
x509-key.pub	The intermediate certificate public key	No use

References

AWX PR for implementing SSH signed key

Red Hat doc on signing SSH key

Using HashiCorp Vault

Extracting SSH pub key from SSL certificate

How Facebook does it