Etcd Encryption at Rest & Basic Control Plane Hardening

Duration: ~25 minutes (most of that is waiting if the cluster still has to encrypt)

Overview

Prove why etcd encryption at rest matters: copy a known Secret from etcd (the store an attacker with a backup would see), enable encryption if it is off, then copy the same key again and show the value is ciphertext.

The Kubernetes API still returns Secrets after encryption. What changes is a stolen etcd snapshot or disk: the planted password must not appear in the raw value.

Why it matters

Etcd holds Secrets, ConfigMaps, Routes, and OAuth tokens. By default OpenShift does not encrypt those values at rest. If someone gets an etcd backup or the volume, they can read credentials. Encryption at rest closes that path. You still need RBAC—oc get secret keeps working for people the API already trusts.

What does it solve

  • Secrets readable from etcd disk or backups if the store is accessed

  • Gaps against common data-protection audit asks

  • Weak control-plane posture when etcd is treated as "just another volume"

Your Mission

If an attacker (or stolen backup) can read raw etcd, unencrypted Secrets become free credentials. Plant a sentinel password, copy that key from etcd, turn on encryption if needed, then copy it again and confirm the sentinel is gone from the etcd value. Complete Part A, then Part B.

This uses the supported etcdctl client inside an openshift-etcd pod—the same check Red Hat documents for verifying encryption. Do not copy other namespaces’ Secrets.

Click each step only if you need a hint.

List nodes and control-plane pods

etcd and the Kubernetes API must be Ready before you plant a Secret or read etcd.

oc get nodes -o wide
oc get pods -n openshift-kube-apiserver
oc get pods -n openshift-etcd
etcd and kube-apiserver pods should be Running.

Part A: Test before encryption

Create a lab Secret, then test it twice: through the API, and through etcd.

Create project 101-08-e-etcd and Secret lab-db-creds

The sentinel password is what you will search for in etcd. Use this exact string so the grep in later steps matches.

oc new-project 101-08-e-etcd || oc project 101-08-e-etcd
oc create secret generic lab-db-creds --from-literal=password='ROADSHOW-ETCD-CLEARTEXT-TOKEN' --dry-run=client -o yaml | oc apply -f -
oc get secret lab-db-creds -o jsonpath='{.data.password}' | base64 -d; echo
The API shows the password. That is expected. Etcd encryption does not hide Secrets from oc get.
Read spec.encryption.type on APIServer cluster

Empty or identity means etcd values are stored in the clear. aesgcm or aescbc means encryption is already on.

oc get apiserver cluster -o jsonpath='spec.encryption.type={.spec.encryption.type}{"\n"}'
oc get kubeapiserver -o jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}reason={.reason}{"\n"}message={.message}{"\n"}{end}'
Note the type. You will skip the patch in Part B if it is already aesgcm or aescbc.
Get the Secret password through the API

Authorized oc get must still show the sentinel. Encryption at rest does not hide Secrets from the API.

oc -n 101-08-e-etcd get secret lab-db-creds -o jsonpath='{.data.password}' | base64 -d; echo
You should see ROADSHOW-ETCD-CLEARTEXT-TOKEN.
Grep the Secret value from etcd

This is the stolen-backup view. Use the etcdctl container in an openshift-etcd pod.

ETCD_POD=$(oc get pod -n openshift-etcd --no-headers | awk '/^etcd-/{print $1; exit}')
echo "$ETCD_POD"
oc exec -n openshift-etcd -c etcdctl "$ETCD_POD" -- \
  etcdctl get /kubernetes.io/secrets/101-08-e-etcd/lab-db-creds --print-value-only \
  | grep -aE 'k8s:enc:|ROADSHOW-ETCD-CLEARTEXT-TOKEN' || echo 'FAIL: no etcd match'

Pass if you see one of:

ROADSHOW-ETCD-CLEARTEXT-TOKEN

(not encrypted yet — continue to Part B) or:

k8s:enc:

(already encrypted — skip the patch in Part B, still run the after test).

Part B: Encrypt, then test again

Enabling encryption is cluster-wide and can take 20 minutes or longer. If the before test already showed k8s:enc: in etcd, skip the patch and run the after test.

Do not enable encryption on a shared roadshow cluster unless you own the cluster or the instructor asked you to. One enable is enough for everyone. Do not turn encryption back off in this lab.
Patch APIServer cluster to encryption type aesgcm

Official procedure: set spec.encryption.type on the APIServer object to aesgcm. Skip the patch if type is already aesgcm or aescbc.

TYPE=$(oc get apiserver cluster -o jsonpath='{.spec.encryption.type}')
echo "Current type: '${TYPE}'"
if [[ "${TYPE}" == "aesgcm" || "${TYPE}" == "aescbc" ]]; then
  echo 'Already encrypted -- skip the patch.'
else
  oc patch apiserver cluster --type=merge -p '{"spec":{"encryption":{"type":"aesgcm"}}}'
  echo 'Patch applied. Encryption of existing objects can take 20+ minutes.'
fi
One enable is enough for the shared cluster. Do not turn encryption back off.
Read Encrypted conditions on the API servers

Re-run this block until you see EncryptionCompleted and secrets, configmaps. EncryptionInProgress means wait and click again.

oc get openshiftapiserver -o jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}{end}'
oc get kubeapiserver -o jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}{end}'
oc get authentication.operator.openshift.io -o jsonpath='{range .items[0].status.conditions[?(@.type=="Encrypted")]}{.reason}{"\n"}{.message}{"\n"}{end}'
Do not take an etcd backup until encryption has finished, or the snapshot can be only partly encrypted.
Get the Secret through the API after encryption

The API must still show the password for authorized users.

oc -n 101-08-e-etcd get secret lab-db-creds -o jsonpath='{.data.password}' | base64 -d; echo
Still ROADSHOW-ETCD-CLEARTEXT-TOKEN.
Grep the Secret value from etcd after encryption

The etcd value must be ciphertext (k8s:enc:), not the password.

ETCD_POD=$(oc get pod -n openshift-etcd --no-headers | awk '/^etcd-/{print $1; exit}')
oc exec -n openshift-etcd -c etcdctl "$ETCD_POD" -- \
  etcdctl get /kubernetes.io/secrets/101-08-e-etcd/lab-db-creds --print-value-only \
  | grep -aE 'k8s:enc:|ROADSHOW-ETCD-CLEARTEXT-TOKEN' || echo 'FAIL: no etcd match'
Pass is k8s:enc:. If you still see ROADSHOW-ETCD-CLEARTEXT-TOKEN, wait for encryption to finish and retry.
Check Pass Fail

API

prints ROADSHOW-ETCD-CLEARTEXT-TOKEN

empty or error

etcd

prints k8s:enc:

prints ROADSHOW-ETCD-CLEARTEXT-TOKEN (wait and retry)

Debrief

You copied a known Secret out of etcd, enabled at-rest encryption if it was off, and copied the same key again. The API still served the Secret; the etcd value became ciphertext. That is the control against a stolen volume or etcd backup.

What breaks without this:

  • Unencrypted etcd / backups → Secrets readable if storage is stolen

  • Weak control-plane posture → larger blast radius when the API plane is probed

  • Backing up etcd during encryption → a mix of clear and encrypted objects

etcd encryption at rest (aesgcm or aescbc), store encryption keys separately from snapshots, node/API hardening, and audit so you can prove the control is on.

Leave encryption enabled. Disabling it (type: identity) decrypts the store again and is out of scope for this lab.

enable at install when you can; enabling later is cluster-wide and can take 20+ minutes. Performance impact is usually small versus the data-protection win. This does not replace RBAC or 101-06 Secret hygiene.
giphy

Cleanup

Before moving to the next module, run the lab cleanup script. It removes the lab project and the etcd copy files. It does not disable cluster etcd encryption.

cd ~/ocp5-rhacs-showroom
bash setup/lab-cleanup.sh --module 101-08