If you found value in this post, consider following me on X @davidpuplava for more valuable information about Game Dev, OrchardCore, C#/.NET and other topics.
It's that time of year again where my homelab Kubernetes cluster (running microk8s) certificates expire.
I didn't even notice.
I happened to check one of my websites and notices nothing came back.
Early notification when there is a problem in my homelab is a discussion for another time.
Today, I want to run through what I did to get my system back up and running.
My website are down, getting 404 not found or 503 service unavailable.
I first log into my kubernetes master node and check to see if my pods are running.
Note: that k is an alias for kubectl.
$ k get all
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?
Oh boy, now what?
I have a few years experience running a kubernetes cluster and am familiar with expired certificates and what a pain it is to fix them.
I happen to be running a lightweight distribution of kubernetes called microk8s. I found their documentation helpful regarding certificates.
$ sudo microk8s refresh-certs --help
[sudo] password for :
Usage: microk8s refresh-certs [OPTIONS] [CA_DIR]
Replace the CA certificates with the ca.crt and ca.key found in CA_DIR.
Omit the CA_DIR argument and use the '--cert' flag to auto-generate a new CA
or any other certificate.
Options:
-c, --check Check the expiration time of the installed certificates
-e, --cert The certificate to be autogenerated, must be one of ['ca.crt', 'server.crt', 'front-proxy-client.crt']
-u, --undo Revert the last refresh performed
-h, --help Show this message and exit.
$ sudo microk8s refresh-certs -c
The CA certificate will expire in 3276 days.
The server certificate will expire in 364 days.
The front proxy client certificate will expire in -9 days.
Whoops, I can't believe my sites have been down for 9 days.
Here's the fix to renew the expired certs:
First the server.crt.
$ sudo microk8s refresh-certs -e server.crt
Taking a backup of the current certificates under /var/snap/microk8s/6673/certs-backup/
Creating new certificates
Signature ok
subject=C = GB, ST = Canonical, L = Canonical, O = Canonical, OU = Canonical, CN = 127.0.0.1
Getting CA Private Key
Restarting service kubelite.
Restarting service cluster-agent.
Then the front-proxy-client.crt.
$ sudo microk8s refresh-certs -e front-proxy-client.crt
Taking a backup of the current certificates under /var/snap/microk8s/6673/certs-backup/
Creating new certificates
Signature ok
subject=CN = front-proxy-client
Getting CA Private Key
Restarting service kubelite.
Now, I can recheck to make sure everything looks good.
$ sudo microk8s refresh-certs -c
The CA certificate will expire in 3276 days.
The server certificate will expire in 364 days.
The front proxy client certificate will expire in 364 days.
Awesome! Now let's see if I can access my kubernetes components.
$ k get nodes
NAME STATUS ROLES AGE VERSION
kbndev01 Ready <none> 366d v1.26.15
kbndev02 Ready <none> 366d v1.26.15
kbmdev01 Ready <none> 373d v1.26.15
Excellent! Now I try to access one of my sites but still nothing is coming up.
After several different troubleshooting steps of restarting services and rebooting all the nodes, I find this.
$ k get nodes
NAME STATUS ROLES AGE VERSION
kbmdev01 Ready <none> 373d v1.26.15
kbndev01 NotReady <none> 366d v1.26.15
kbndev02 NotReady <none> 366d v1.26.15
Why are these nodes not ready?
Do I need to refresh the certs on them as well?
The answer is yes.
I log into each node individually and run the refresh-certs commands to fix the certs.
Then after a few seconds, all the nodes are ready.
$ k get nodes
NAME STATUS ROLES AGE VERSION
kbmdev01 Ready <none> 375d v1.26.15
kbndev02 Ready <none> 368d v1.26.15
kbndev01 Ready <none> 368d v1.26.15
After that, all my sites were once again accessible.
If your kubernetes certificates are expired, especially when using the microk8s distribution, be sure to renew all the certs on all the nodes.
If you found value in this post, consider following me on X @davidpuplava for more valuable information about Game Dev, OrchardCore, C#/.NET and other topics.