ESXi Certificates using Certbot

Here at $HOME, I have a good infrastructure to perform tests and study whatever I want. Part of that infrastructure is an ESXi host with lots of VMs, which also include my “bare-metal” Kubernetes cluster. Another thing I have around is an internal DNS server, which uses one of my domains to give those VMs different names.

However, there’s one thing that bothers me a lot: self-signed certificates. Some may say it is completely overkill to issue certificates for hosts that aren’t even exposed to the public internet, and they aren’t wrong. The point is that those warnings I get on my browser annoy me to a degree that I rather jump through some hoops than to keep getting them.

The issue is simply solved using the great Certbot to issue a Let’s Encrypt wildcard certificate to that domain I mentioned. That’s something straightforward to do, since everything I need is to have a way to pass its dns-01 challenge, which simply involves writing a small Python script that is called by Certbot, creates TXT records on my DNS provider (the one keeping the public records of that domain), and another one to clean it up afterwards.

However, there’s a small issue there. Certificates generated by Certbot are using elliptic curves, and ESXi isn’t happy with it. Replacing certificates from /etc/vmware/ssl with the ones generated by Certbot makes it quite sad:

Failed to initialize the SSL context: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:0688010A:asn1 encoding routines::nested asn1 error)

For ESXi, we will need to ask for an RSA certificate instead. This post is more of a reminder, cause I always forget what I did the last time that certificate expired, and immediately enter on a loop of “what is going on here?!”

First, ask certbot to generate a certificate using RSA:

certbot certonly --manual \
                 --preferred-challenges=dns \
                 --manual-auth-hook /var/dns/script/authenticator.sh \
                 --manual-cleanup-hook /var/dns/script/cleanup.sh \
                 -d '$ESXI_HOST' \
                 --key-type rsa

Then, ensure the ESXi host have SSH enabled, and:

scp /etc/letsencrypt/live/$ESXI_HOST/fullchain.pem user@$ESXI_HOST:/etc/vmware/ssl/rui.crt
scp /etc/letsencrypt/live/$ESXI_HOST/privkey.pem user@$ESXI_HOST:/etc/vmware/ssl/rui.key
ssh user@$ESXI_HOST -C "services.sh restart"

In case of emergencies, remember that the self-signed certificates can be restored using generate-certificates:

ssh user@$ESXI_HOST -C "/sbin/generate-certificates"

Of course, remember to restart the host services:

ssh user@$ESXI_HOST -C "services.sh restart"

Renewing the certificate is just a matter to re-run the certbot cert-only command (in full), copying fullchain.pem and privkey.pem to the host, and restarting its services. Of course this can be scripted as a cron-job. Here I have a small PC Engines APU that is responsible for running the DNS server, that will eventually have a cron-job for doing that, but before that I intend to keep renewing manually once or twice to make sure everything goes well.

An attention point here when renewing is that Certbot will suggest to upgrade the certificate to use elliptic curve. Do not upgrade it. Upgrading will require the certificate to be destroyed and re-created using RSA.

Finally, if you got at this point and also leverages Let’s Encrypt on your public or private hosts, consider donating to the Electronic Frontier Foundation, and support them on their mission to defend privacy, free speech, civil liberties and human rights online.