We establish an emergency access procedure for SSH hosts with hardware keys

In this post, we will develop a procedure for emergency access to SSH hosts using hardware security keys offline. This is just one approach, and you can adapt it to your needs. We will store the SSH certification authority for our hosts on the hardware security key. This scheme will work on virtually any OpenSSH, including SSH with single sign-on.

Why all this? Well, it's an option for emergencies. It's a backdoor that allows you to access your server when, for some reason, nothing else works.

Why use certificates instead of public/private keys for emergency access?

• Unlike public keys, certificates can have a very short validity period. You can generate a certificate valid for 1 minute or even 5 seconds. Once that time elapses, the certificate becomes unusable for new connections. This is perfect for emergency access.
• You can create a certificate for any account on your hosts and send such "one-time" certificates to colleagues when necessary.

What You'll Need

• Hardware security keys that support resident keys.
  Resident keys are cryptographic keys that are fully stored inside the security key. Sometimes they are protected by an alphanumeric PIN. The public part of the resident key can be exported from the security key, if necessary, along with the private key descriptor. Resident key support is available in, for example, Yubikey 5 USB keys. It is recommended that they are intended solely for emergency access to the host. For this post, I will use only one key, but you should have an additional one for backup.
• A secure place to store these keys.
• OpenSSH version 8.2 or higher on your local computer and on the servers you wish to access for emergencies. Ubuntu 20.04 comes with OpenSSH 8.2.
• (optional but recommended) A CLI tool for verifying certificates.

Preparation

First, you need to create a certification authority, which will be on the hardware security key. Insert the key and run:

$ ssh-keygen -t ecdsa-sk -f sk-user-ca -O resident -C [security key ID]

As a comment (-C), I specified yubikey-9-512-742@smallstep.com to remember which security key this certificate authority refers to.

In addition to adding a key to Yubikey, two files will be generated locally:

1. sk-user-ca, a key descriptor that refers to the private key stored in the security key,
2. sk-user-ca.pub, which will be the public key for your certificate authority.

But don't worry, another private key is stored on Yubikey that cannot be extracted. So everything is secure here.

On hosts as the user root, add (if not already added) the following to your SSHD configuration (/etc/ssh/sshd_config):

TrustedUserCAKeys /etc/ssh/ca.pub

Then on the host, add the public key (sk-user-ca.pub) to /etc/ssh/ca.pub

Restart the daemon:

# /etc/init.d/ssh restart

Now we can try to access the host. But first, we need a certificate. Create a key pair that will be associated with the certificate:

$ ssh-keygen -t ecdsa -f emergency

Certificates and SSH pairs
Sometimes there's a temptation to use a certificate as a substitute for a public/private key pair. But for user authentication, one certificate is not enough. Each certificate also has a private key associated with it. That's why we need to generate this pair of 'emergency' keys before we issue ourselves a certificate. It is important that we show the signed certificate to the server, specifying the key pair for which we have the private key.

Thus, the exchange of public keys is still very much alive. It even works with certificates. Certificates simply relieve the server of the need to store public keys.

Next, create the certificate itself. I need user ubuntu to be authorized within a 10-minute interval. You can do it your own way.

$ ssh-keygen -s sk-user-ca -I test-key -n ubuntu -V -5m:+5m emergency

You will be prompted to sign the certificate using the fingerprint. You can add additional usernames separated by commas, like -n ubuntu,carl,ec2-user

That's it, now you have a certificate! Next, set the correct permissions:

$ chmod 600 emergency-cert.pub

After that, you can view the contents of your certificate:

$ step ssh inspect emergency-cert.pub

Here’s what my certificate looks like:

emergency-cert.pub
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:EJSfzfQv1UK44/LOKhBbuh5oRMqxXGBSr+UAzA7cork
        Signing CA: SK-ECDSA SHA256:kLJ7xfTTPQN0G/IF2cq5TB3EitaV4k3XczcBZcLPQ0E
        Key ID: "test-key"
        Serial: 0
        Valid: from 2020-06-24T16:53:03 to 2020-06-24T17:03:03
        Principals:
                ubuntu
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

Here, the public key is the emergency key we created, and it is associated with the sk-user-ca certification authority.

Finally, we are ready to execute the SSH command:

$ ssh -i emergency ubuntu@my-hostname
ubuntu@my-hostname:~$

1. Now you can create certificates for any user on the host that trusts your certificate authority.
2. You can delete emergency. You can keep sk-user-ca, but you don’t need it since it's also stored on the security key. You might also want to remove the original PEM public key from your hosts (e.g., in ~/.ssh/authorized_keys for the ubuntu user) if you used it for emergency access.

Emergency Access: Action Plan

Insert the security key and run the command:

$ ssh-add -K

This will add the public key of the certification authority and the key descriptor to the SSH agent.

Now export the public key to create the certificate:

$ ssh-add -L | tail -1 > sk-user-ca.pub

Create a time-limited certificate, for instance, valid for no more than an hour:

$ ssh-keygen -t ecdsa -f emergency
$ ssh-keygen -Us sk-user-ca.pub -I test-key -n [username] -V -5m:+60m emergency
$ chmod 600 emergency-cert.pub

And now again SSH:

$ ssh -i emergency username@host

If your .ssh/config file causes any issues when connecting, you can run ssh with the -F none option to bypass it. If you need to send the certificate to a colleague, the simplest and most secure way is to use Magic Wormhole. For this, you will need just two files — in our case, these are emergency and emergency-cert.pub.

What I like about this approach is the hardware support. You can keep your security keys in a safe, and they won’t go anywhere.

Advertising

Epic servers — this is Affordable VPS with powerful AMD processors, CPU core frequency up to 3.4 GHz. The maximum configuration can handle almost any task — 128 CPU cores, 512 GB RAM, 4000 GB NVMe. Join us!

Source: habr.com