Secrets Without Static Copies – Vault CSI Integration

Table of Contents

Overview
Why it matters
What does it solve
Your Mission
Debrief
Cleanup

Duration: ~25 minutes

Overview

Wire HashiCorp Vault and External Secrets Operator so pods get secrets at runtime without baking static copies into Git or images—dev-mode Vault for learning only.

Use the consoles and credentials from Home for this exercise.

OpenShift console

{openshift_cluster_console_url}

OpenShift username

{openshift_cluster_admin_username}

OpenShift password

{openshift_cluster_admin_password}

RHACS console

https://central-reencrypt-stackrox.{openshift_cluster_ingress_domain}

RHACS username

admin

RHACS password

{common_password}

Why it matters

Static credentials in images, manifests, and long-lived Kubernetes Secrets are prime loot after a foothold. Vault keeps the source of truth off the cluster disk and issues access on demand so a stolen YAML file is no longer enough.

What does it solve

Secrets committed to Git or baked into images
Long-lived cluster Secrets with no rotation story
Apps that cannot prove where credentials came from

Your Mission

Stand up Vault, configure Kubernetes auth and policies, then prove a pod can consume secrets without carrying static copies—an attacker who clones the app repo still does not get the database password.

This lab runs Vault in dev mode (auto-unsealed, root token root). Dev mode is for learning only—data is lost when the Vault pod restarts. Do not use it in production.

This lab tracks the Layered Zero Trust Validated Pattern pattern for secrets (Vault and External Secrets Operator).

Click each step only if you need a hint.

Run install-vault.sh

The installer lives in this roadshow repo. It deploys the HashiCorp Helm chart into vault with the Agent Injector enabled (dev mode, root token root).

cd ~/openshift-security-roadshow/setup/vault-lab
./install-vault.sh

CTRL+click the URL the script prints to open the Vault UI.

Get pods, services, and the Route in namespace vault

Confirm the server and injector are up before you exec in.

oc get pods,svc,route -n vault

vault-0 and vault-agent-injector are Running, and a Route exists.

Exec into vault-0

Stay in this shell for the next Vault CLI steps.

oc exec -it -n vault vault-0 -- /bin/sh

Your prompt is now inside vault-0.

Set VAULT_ADDR and VAULT_TOKEN inside the pod

Dev-mode token is root. Paste these in the vault-0 shell.

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root

vault status should succeed after this.

Write /home/vault/read-policy.hcl

The policy only allows read on secret*.

cat <<EOF > /home/vault/read-policy.hcl
path "secret*" {
  capabilities = ["read"]
}
EOF

Run this inside the Vault pod shell from the previous step.

Write Vault policy read-policy

Apply the HCL file you just created.

vault policy write read-policy /home/vault/read-policy.hcl

Success! Uploaded policy: read-policy.

Enable Kubernetes auth

Pods will authenticate with their SA token, not the root token.

vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/.

Write auth/kubernetes/config

Point Vault at this cluster’s API using the pod’s SA token and CA.

vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Success! Data written to: auth/kubernetes/config.

Enable KV v2 at path secret/

Ignore the error if the engine is already on.

vault secrets enable -path=secret kv-v2 2>/dev/null || true

Engine is available at secret/.

Create Kubernetes auth role vault-role

Bind policy read-policy to SA vault-serviceaccount in namespace vault. Vault 1.21 checks the token aud claim.

AUD=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d. -f2 | base64 -d 2>/dev/null | sed -n 's/.*"aud":\["\([^"]*\)".*/\1/p')
vault write auth/kubernetes/role/vault-role \
  bound_service_account_names=vault-serviceaccount \
  bound_service_account_namespaces=vault \
  policies=read-policy \
  ttl=1h \
  audience="${AUD}"

Success! Data written to: auth/kubernetes/role/vault-role.

Exit the Vault pod shell

Leave vault-0 before you run oc commands again.

exit

Your prompt is the bastion again.

Exec into vault-0 again

Reconnect if you already exited.

oc exec -it -n vault vault-0 -- /bin/sh

You are inside vault-0 again.

Set VAULT_ADDR and VAULT_TOKEN again

Paste these in the vault-0 shell.

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root

Ready to run vault kv commands.

Put secret/login in Vault

This path is what the Agent Injector will mount later.

vault kv put secret/login rhos-token=workshop-token-abc123

created / version metadata prints.

Put secret/my-first-secret in Vault

Username and password for the first sample app.

vault kv put secret/my-first-secret username=workshop-user password=workshop-password

A new KV version is stored.

List secrets under secret/

Confirm both paths exist.

vault kv list secret

List includes login and my-first-secret.

Exit the Vault pod shell after creating secrets

exit

Back on the bastion.

Print the Vault UI URL

Open the URL in a new tab (Ctrl+click or Cmd+click).

echo "https://$(oc get route vault -n vault -o jsonpath='{.spec.host}')/"

Create secret/my-second-secret in the Vault UI

ESO will sync this path into a Kubernetes Secret. The names are different on purpose.

Navigate to Secrets Engines → secret.
Click Create secret.
Path: my-second-secret
Keys: username = workshop-user-2, password = workshop-password-2
Click Save.

The Vault path must be exactly my-second-secret (under the secret engine). ESO writes Kubernetes Secret my-second-secret-sync.

Apply vault-secrets-app.yaml

The Agent Injector mutates the pod and writes files under /vault/secrets/.

oc apply -f ~/openshift-security-roadshow/setup/vault-lab/vault-secrets-app.yaml

Deployment and Route are created in namespace vault.

Watch vault-secrets-app pods

Wait for 2/2 Ready (app + Vault Agent sidecar). Press Ctrl+C when it is Ready.

oc get pods -n vault -l app=vault-secrets-app -w

2/2 Running.

Print the vault-secrets-app Route

Open the app in a new tab.

echo "https://$(oc get route vault-secrets-app -n vault -o jsonpath='{.spec.host}')/"

You should see the Vault Secrets App welcome page.

Set POD to the vault-secrets-app pod name

POD=$(oc get pod -n vault -l app=vault-secrets-app -o jsonpath='{.items[0].metadata.name}')
echo "$POD"

A pod name prints. Re-run this if you open a new terminal.

Cat injected files from the web container

Read /vault/secrets/login and /vault/secrets/my-first-secret.

oc exec -n vault "${POD}" -c web -- sh -c "cat /vault/secrets/login && echo && cat /vault/secrets/my-first-secret"

Expect rhos-token=workshop-token-abc123 and username=workshop-user / password=workshop-password.

Read vault-agent-init logs if the mount is empty

Use the same POD variable from the previous step, or set it again.

oc logs -n vault "${POD}" -c vault-agent-init

Auth and render errors show up here if files are missing.

Add the external-secrets Helm repo

helm repo add external-secrets https://charts.external-secrets.io

external-secrets is in your Helm repo list.

Update the external-secrets Helm repo

helm repo update external-secrets

Charts are current.

Helm install External Secrets Operator

ESO copies Vault data into native Kubernetes Secrets.

helm upgrade --install external-secrets external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true \
  --wait --timeout 10m

Helm reports a successful install.

Get pods in namespace external-secrets

oc get pods -n external-secrets

The external-secrets pod is Running.

Apply eso-secretstore.yaml

Creates the token Secret, SecretStore, and ExternalSecret that read my-second-secret.

oc apply -f ~/openshift-security-roadshow/setup/vault-lab/eso-secretstore.yaml

SecretStore and ExternalSecret exist in namespace vault.

Get ExternalSecret and SecretStore in vault

oc get externalsecret,secretstore -n vault

sync-my-second-secret is listed.

Check ExternalSecret Ready status

oc get externalsecret sync-my-second-secret -n vault -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'

True. If SecretSyncedError, confirm my-second-secret in the Vault UI has both keys.

Decode username from Secret my-second-secret-sync

oc get secret my-second-secret-sync -n vault -o jsonpath='{.data.username}{"\n"}' | base64 -d && echo

Decoded username is workshop-user-2.

Apply eso-vault-test.yaml

This app reads env from the synced Kubernetes Secret, not Agent files.

oc apply -f ~/openshift-security-roadshow/setup/vault-lab/eso-vault-test.yaml

Deployment is created in namespace vault.

Watch eso-read-vault-secret pods

Press Ctrl+C when the pod is 1/1 Ready.

oc get pods -n vault -l app=eso-read-vault-secret -w

1/1 Running.

Set ESO_POD to the eso-read-vault-secret pod name

ESO_POD=$(oc get pod -n vault -l app=eso-read-vault-secret -o jsonpath='{.items[0].metadata.name}')
echo "$ESO_POD"

A pod name prints.

Print USERNAME and PASSWORD from the ESO app pod

oc exec -n vault "${ESO_POD}" -c app -- sh -c 'echo "username=${USERNAME}" && echo "password=${PASSWORD}"'

username=workshop-user-2 and password=workshop-password-2.

Debrief

You consumed Vault secrets two ways—Agent Injector files under /vault/secrets/ and External Secrets Operator sync into a native Secret—so credentials are not baked into Git or images.

What breaks without this:

Static Secrets in etcd/Git/images → long-lived loot after a foothold
No rotation path → stolen credentials stay valid

Vault (or equivalent) as source of truth, Kubernetes auth bound to namespace/SA, injector for file delivery, ESO when apps need native Secret mounts, and least-privilege Vault roles.

prefer short-lived, identity-bound access over copied passwords. Lab cleanup runs setup/vault-lab/cleanup-vault-lab.sh via lab-cleanup.sh --module 201-05.

Cleanup

When you finish this module, run the roadshow cleanup script. It uninstalls External Secrets Operator and Vault and removes the related namespaces.

cd ~/openshift-security-roadshow
bash setup/lab-cleanup.sh --module 201-05