Creating Client Certificates for Kubernetes Pods

An interesting challenge came up an work recently about how to do TLS Client Authentication from Kubernetes pods with short lived certificates.

The main gist was how to dynamically update the certificates on a short time frame (e.g. once a week) but without needing to manually mess about with recreating the Secrets that hold the keys and certificates

Cert-Manager

The standard tool for dealing with Certificate Authorities in Kubernetes is cert-manager, the usual use case is to issue server certificates either linked to an Ingress objects by adding annotations to these objects. These can then be provisioned from the CA, usually by using ACME. I set up a Small Step CA with ACME to issue certificates on my LAN.

This basically works by creating a Kubernetes Web Hook that looks for when Ingress are created or modified and then reading the annotation and creating a Certificate Signing Request which is then submitted to the CA for signing and the resultant Certificate and Key are stored in a Kubernetes Secret.

Cert-Manager-CSI

This allows for the creation of Certificates for things that are not exposed by Ingress e.g. where the TLS termination is being done in the Pod rather than be delegated to the HTTP reverse proxy providing the Ingress service or for Client Certificates.

The CSI driver allows virtual volumes to be created that mount the Secret that holds a x509 Certificate and Private Key to be mounted as files into a Pod.

When I first installed the CSI driver I couldn’t get it to work as it kept complaining that /var/snap was a Read Only file system, this turned out to be because I use microk8s as my Kubernetes distribution and this has a none standard kubelet directory location because it is packaged as a snap bundle. The solution is to pass the correct kubelet path to the helm chart

$ helm upgrade cert-manager-csi-driver oci://quay.io/jetstack/charts/cert-manager-csi-driver \

--install \

--namespace cert-manager \

--set app.kubeletRootDir=/var/snap/microk8s/common/var/lib/kubelet \

--wait

Small Step Certificate Issuer

As I already have a Small Step CA up and running to provide server certificates via ACME I needed to add a new provisioner that would create client certificates with the correct flags. I could probably have reused to the one I set up to do email signing but I decided to keep them separate for this test.

Details of how to install and configure the required tools can be found here.

$ step ca provisioner add "step-client-certs" --type JWK --create \

--x509-min-dur 24h --x509-max-dur 720h --x509-default-dur 720h

Keep a note of the password created as part of this process as it will need to be added to a Secret later, the kid of the provisioner is also required.

$ kubectl create secret -n step-issuer generic step-client-certs-password --from-literal=password=TopSecret

The Step Issuer is installed with

$ helm install \

step-issuer smallstep/step-issuer \

--namespace step-issuer \

--create-namespace

Then either a StepIssuer or a StepClusterIssuer I’m going to user a Cluster scoped Issuer so I don’t have to create one in every namespace I’m using.

apiVersion: certmanager.step.sm/v1beta1

kind: StepClusterIssuer

metadata:

namespace: step-issuer

spec:

url: https://certs.loc:9000

caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJwekNDQVU2Z0F3SUJBZ0lSQUp1MnByWWJFa09HRGRBSnpORXErYWt3Q2dZSUtvWkl6ajBFQXdJd01qRVQKTUJFR0ExVUVDaE1LU0dGeVpHbHNiQ0JEUVRFYk1Ca0dBMVVFQXhNU1NHRnlaR2xzYkNCRFFTQlNiMjkwSUVOQgpNQjRYRFRJeU1USXdOREUxTWpVME1sb1hEVE15TVRJd01URTFNalUwTWxvd01qRVRNQkVHQTFVRUNoTUtTR0Z5ClpHbHNiQ0JEUVRFYk1Ca0dBMVVFQXhNU1NHRnlaR2xzYkNCRFFTQlNiMjkwSUVOQk1Ga3dFd1lIS29aSXpqMEMKQVFZSUtvWkl6ajBEQVFjRFFnQUVPRXI2ZFdZL1JjdStHNzBUV0FVbWVSVHQycFg1cEpwTy9YNkJHSHM3NkdlaQpWcFhGeTZ1YXFFeHVCc2xIdGRiMGtUanNkbWlnM2l5ckhFQmhXRVFCd3FORk1FTXdEZ1lEVlIwUEFRSC9CQVFECkFnRUdNQklHQTFVZEV3RUIvd1FJTUFZQkFmOENBUUV3SFFZRFZSME9CQllFRkhSOG1nUndMUGN0UkZmV3B6Rk0KMk5mMDFkT2RNQW9HQ0NxR1NNNDlCQU1DQTBjQU1FUUNJSDc1bUh2cUxsRDRmYy94ZmFCZjVpc2htRWVJZW4xQQpiVEU5VTZxODRnZHdBaUJUVEZaWjB1dzJBSlpmQ1ZuWmFSQ3hmYnhzYURZNFBSQTNaMjdTczBkd2FRPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

provisioner:

kid: IdYupAKmyDlfKPQlDP5IAqVhzU3eC0Au0OUj1wxDeTY

passwordRef:

key: password

namespace: step-issuer

Testing it all

I have a standard bare Ubuntu pod I tend to use for debugging when I need a shell in the cluster. I modified it to request a client certificate and mount the files on /tls

apiVersion: v1

kind: Pod

metadata:

labels:

app: ubuntu

spec:

containers:

- image: "ubuntu:24.04"

command:

- "tail"

- "-f"

- "/dev/null"

imagePullPolicy: IfNotPresent

volumeMounts:

- name: tls

mountPath: /tls

volumes:

- name: tls

csi:

driver: csi.cert-manager.io

readOnly: true

volumeAttributes:

csi.cert-manager.io/issuer-name: step-client-issuer

csi.cert-manager.io/issuer-group: certmanager.step.sm

csi.cert-manager.io/issuer-kind: StepClusterIssuer

csi.cert-manager.io/common-name: ${POD_NAME}-${POD_NAMESPACE}

csi.cert-manager.io/key-usages: digital signature,client auth

csi.cert-manager.io/duration: 720h

restartPolicy: Always

The important configuration parts as the volumeAttributes which set which Issuer to use and what Common Name to set.

$ kubectl exec -it ubuntu -- /bin/bash

root@ubuntu:/# cd /tls

root@ubuntu:/tls# ls

ca.crt tls.crt tls.key

root@ubuntu:/tls# cat tls.crt

-----BEGIN CERTIFICATE-----

MIIDADCCAqegAwIBAgIQScv/Sg70g2nvMynmAej51jAKBggqhkjOPQQDAjA6MRMw

EQYDVQQKEwpIYXJkaWxsIENBMSMwIQYDVQQDExpIYXJkaWxsIENBIEludGVybWVk

...