Kubernetes Security and PKI

Kubernetes has a built-in PKI that secures all internal cluster communication. A cluster Certificate Authority (CA) signs certificates for every component: the API server, kubelets, etcd nodes, controller-manager, scheduler, and front-proxy. These certificates provide mutual authentication — every component proves its identity to every other component via TLS client certificates. RBAC authorization decisions are based on the identity in these certificates (Common Name = username, Organization = group). The cluster CA is the root of all trust within the cluster.

Why it matters

• All authentication is certificate-based — kubelets authenticate to the API server with client certificates. The API server authenticates to etcd with client certificates. Without valid certificates, components can’t communicate.
• RBAC identity from certificates — when a kubelet connects, its certificate CN (system:node:node-01) and O (system:nodes) determine its RBAC permissions. A certificate with CN=system:masters has cluster-admin access.
• etcd encryption — etcd stores all cluster state (Secrets, ConfigMaps, RBAC policies). etcd peer and client certificates ensure only authorized components can read or write cluster state.
• Cluster CA is the crown jewel — anyone with the cluster CA private key can issue certificates for any identity, granting any RBAC role. It’s equivalent to permanent, undetectable cluster-admin access.
• Certificate rotation — cluster certificates expire (1 year for kubeadm). Rotation must happen without downtime, which requires careful coordination between control plane components.

How it works

Cluster PKI hierarchy:

1. Cluster CA — self-signed root that signs all cluster certificates. Generated at cluster creation (/etc/kubernetes/pki/ca.crt and ca.key).
2. API server certificate — server cert for the API server (SANs include: kubernetes, kubernetes.default, cluster IP, node IPs, external DNS)
3. API server client certs — API server uses these to authenticate to kubelet and etcd
4. Kubelet certificates — each node has a client cert (authenticates to API server) and server cert (serves metrics/exec endpoints)
5. etcd CA — separate CA for etcd (optional but recommended). Signs etcd peer and client certificates.
6. Front-proxy CA — signs certificates for the API aggregation layer (metrics-server, custom API servers)
7. Service account key pair — signs and verifies service account tokens (JWT)

Authentication flow:

kubectl → (presents client cert) → API Server
API Server → (presents client cert) → etcd
API Server → (presents client cert) → Kubelet
Kubelet → (presents client cert) → API Server

In real systems

Cluster PKI files (kubeadm):

/etc/kubernetes/pki/
├── ca.crt                    # Cluster CA certificate
├── ca.key                    # Cluster CA private key (PROTECT THIS)
├── apiserver.crt             # API server serving certificate
├── apiserver.key             # API server private key
├── apiserver-kubelet-client.crt  # API server → kubelet auth
├── apiserver-etcd-client.crt     # API server → etcd auth
├── front-proxy-ca.crt        # Aggregation layer CA
├── front-proxy-client.crt    # Aggregation layer client cert
├── etcd/
│   ├── ca.crt               # etcd CA
│   ├── server.crt           # etcd server cert
│   ├── peer.crt             # etcd peer cert
│   └── healthcheck-client.crt
└── sa.key / sa.pub          # Service account signing key pair

Kubelet certificate rotation (automatic):

# kubelet config (/var/lib/kubelet/config.yaml)
rotateCertificates: true
# Kubelet automatically requests new certificates before expiry
# Requires: API server has CSR approval enabled

# Check kubelet certificate
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate

Creating a user certificate for kubectl:

# Generate key and CSR
openssl genrsa -out developer.key 2048
openssl req -new -key developer.key -out developer.csr -subj "/CN=alice/O=developers"

# Submit CSR to Kubernetes
cat <<EOF | kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: alice-csr
spec:
  request: $(cat developer.csr | base64 | tr -d '\n')
  signerName: kubernetes.io/kube-apiserver-client
  usages: ["client auth"]
EOF

# Approve and retrieve
kubectl certificate approve alice-csr
kubectl get csr alice-csr -o jsonpath='{.status.certificate}' | base64 -d > developer.crt

# CN=alice maps to username "alice" in RBAC
# O=developers maps to group "developers" in RBAC

Where it breaks

Cluster CA key exposed — the CA key (/etc/kubernetes/pki/ca.key) is readable by root on control plane nodes. If a control plane node is compromised, the attacker has the CA key and can issue certificates for CN=system:masters (cluster-admin). There’s no revocation mechanism for cluster certificates — the only remediation is rotating the entire cluster CA (which requires updating every node’s trust). Protect control plane nodes as the most critical infrastructure in your environment.

Kubelet certificate rotation disabled — kubelet certificates expire after 1 year (kubeadm default). If rotateCertificates: true isn’t set, the kubelet’s certificate expires and the node becomes NotReady. The kubelet can’t authenticate to the API server, pods on that node are evicted, and the node is effectively dead until manually re-enrolled. Always enable automatic rotation.

etcd accessible without client certificate — etcd is configured to listen on 0.0.0.0:2379 without requiring client certificate authentication. Any pod or process that can reach the etcd port can read all cluster secrets (including TLS private keys stored as Secrets). etcd must require client certificates (--client-cert-auth=true) and listen only on the control plane network interface.

Operational insight

The Kubernetes cluster CA has no expiry monitoring by default, no rotation automation, and no revocation mechanism. If the CA certificate expires (10 years for kubeadm, but custom clusters may differ), the entire cluster stops functioning — and unlike component certificates, you can’t renew the CA through normal cluster operations because the cluster itself is broken. For self-managed clusters: monitor the CA certificate expiry (it’s just a file — check with openssl x509 -enddate), plan CA rotation well in advance (it requires updating every node’s trust bundle), and consider using an external CA (like Vault) as the cluster’s signing authority for better lifecycle management.

Related topics

• TLS Certificates in Kubernetes
• What is cert-manager
• What is mTLS
• Root CA vs Intermediate CA
• PKI Hierarchy Design