Secrets Without Static Copies – Vault CSI Integration

Table of Contents

Overview
Why it matters
What does it solve
Your Mission
Debrief
Cleanup

Duration: ~25 minutes

Overview

Wire HashiCorp Vault and External Secrets Operator so pods get secrets at runtime without baking static copies into Git or images—dev-mode Vault for learning only.

Unresolved include directive in modules/ROOT/pages/201-06-vault-csi.adoc - include::partial$console-access.adoc[]

Why it matters

Static credentials in images, manifests, and long-lived Kubernetes Secrets are prime loot after a foothold. Vault keeps the source of truth off the cluster disk and issues access on demand so a stolen YAML file is no longer enough.

What does it solve

Secrets committed to Git or baked into images
Long-lived cluster Secrets with no rotation story
Apps that cannot prove where credentials came from

Your Mission

Stand up Vault, configure Kubernetes auth and policies, then prove a pod can consume secrets without carrying static copies—an attacker who clones the app repo still does not get the database password.

This lab runs Vault in dev mode (auto-unsealed, root token root). Dev mode is for learning only—data is lost when the Vault pod restarts. Do not use it in production.

This lab tracks the Layered Zero Trust Validated Pattern pattern for secrets (Vault and External Secrets Operator).

Click each step only if you need a hint.

Deploy Vault

The workshop install script deploys the HashiCorp Helm chart into the vault namespace with the Agent Injector enabled.

Procedure

Clone the repo and run the installer from the bastion:

cd ~/
git clone https://github.com/rhpds/openshift-days-ops-showroom.git
cd openshift-days-ops-showroom/setup-scripts/vault-config-scripts
./install-vault.sh

The script:

Adds the HashiCorp Helm repository and installs chart hashicorp/vault
Enables OpenShift integration and dev mode
Creates an OpenShift Route for the Vault UI
Enables the Vault Agent Injector for pod secret injection

CTRL + Click on the URL after the script completes to open the Vault UI in a new tab.

Verify the deployment

Confirm the Vault server and injector are running:

oc get pods,svc,route -n vault

You should see the Vault server pod Running, the vault-agent-injector deployment Running, and a Route to the UI.

Configure Vault

Configure Vault policies and Kubernetes auth so only authorized pods in namespace vault can read secrets. Run the following inside the Vault pod, then verify in the UI.

Procedure

Connect to the Vault pod:

oc exec -it -n vault vault-0 -- /bin/sh
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root

Create a policy file that allows a pod to read secrets:

cat <<EOF > /home/vault/read-policy.hcl
path "secret*" {
  capabilities = ["read"]
}
EOF

Apply the policy:

vault policy write read-policy /home/vault/read-policy.hcl

Next, enable Kubernetes authentication:
```
vault auth enable kubernetes
```

And allow Vault to communicate with the Kubernetes API server via a service account token:

vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://kubernetes.default.svc:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Enable the KV v2 secrets engine at secret/:

vault secrets enable -path=secret kv-v2 2>/dev/null || true

Create role vault-role that binds the policy to service account vault-serviceaccount in namespace vault:

AUD=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token | cut -d. -f2 | base64 -d 2>/dev/null | sed -n 's/.*"aud":\["\([^"]*\)".*/\1/p')
vault write auth/kubernetes/role/vault-role \
  bound_service_account_names=vault-serviceaccount \
  bound_service_account_namespaces=vault \
  policies=read-policy \
  ttl=1h \
  audience="${AUD}"

Vault 1.21 validates the service account token aud claim against the role audience. The command above reads the cluster audience from the Vault pod token (typically https://kubernetes.default.svc on OpenShift).

Exit the Vault pod shell when configuration is complete:
```
exit
```

Create secrets in Vault

We’ve created some roles. Now, let’s create some secrets in Vault.

Procedure

Reconnect to the Vault pod if you exited the shell:

oc exec -it -n vault vault-0 -- /bin/sh
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=root

Create and verify secrets:

vault kv put secret/login rhos-token=workshop-token-abc123
vault kv put secret/my-first-secret username=workshop-user password=workshop-password
vault kv list secret

Expected: vault kv list secret shows login and my-first-secret.

Exit the Vault pod shell:
```
exit
```

=== Verify the secrets in the Vault UI

Procedure

Print the Vault UI URL, then open it in a new browser tab (Ctrl+click on Windows/Linux, Cmd+click on Mac):
```
echo "https://$(oc get route vault -n vault -o jsonpath='{.spec.host}')/"
```
Sign in with method Token and token root. You should see your newly created secrets in the UI.
Navigate to Secrets Engines → secret.

Click Create secret and set:

Path: my-second-secret
Secret data: add two key/value pairs:

username = workshop-user-2

password = workshop-password-2

The Vault path must be exactly my-second-secret (under the secret engine). ESO reads from that path and writes a Kubernetes Secret named my-second-secret-sync—the names are different on purpose.

Review the secret information and click Save.

Access secrets from a Kubernetes pod

With Vault configured and secrets stored, you will deploy a sample application in the vault namespace. The Vault Agent Injector mutates the pod to add an init container that authenticates with Kubernetes auth and renders secrets to /vault/secrets/.

The deployment uses annotations to inject secret/login and secret/my-first-secret:

vault.hashicorp.com/agent-inject: "true" — enable injection for this pod
vault.hashicorp.com/role: "vault-role" — Kubernetes auth role to use
vault.hashicorp.com/agent-inject-secret-<name> — Vault path to fetch
vault.hashicorp.com/agent-inject-template-<name> — format written to the volume

Procedure

Deploy the Vault Secrets App:

oc apply -f ~/openshift-security-roadshow/setup/vault-lab/vault-secrets-app.yaml

Wait for the pod to be ready:
```
oc get pods -n vault -l app=vault-secrets-app -w
```
Press Ctrl+C when the pod is 2/2 Ready (web container + Vault Agent sidecar).
Open the Vault Secrets App in your browser (Ctrl+click on Windows/Linux, Cmd+click on Mac):
```
echo "https://$(oc get route vault-secrets-app -n vault -o jsonpath='{.spec.host}')/"
```
You should see the Vault Secrets App welcome page.

Read the injected secrets from the pod:

POD=$(oc get pod -n vault -l app=vault-secrets-app -o jsonpath='{.items[0].metadata.name}')
oc exec -n vault "${POD}" -c web -- sh -c "cat /vault/secrets/login && echo && cat /vault/secrets/my-first-secret"

Expect rhos-token=workshop-token-abc123 and username=workshop-user / password=workshop-password.

Confirm the Agent Injector rendered the files (and gather evidence if the mount is empty):
```
oc logs -n vault "${POD}" -c vault-agent-init
oc describe pod -n vault "${POD}"
```

You have injected the first secret into a running pod. Next, use External Secrets Operator to sync the second secret into another workload.

Access secrets with External Secrets Operator

The first deployment used the Vault Agent Injector to render secrets as files under /vault/secrets/. The External Secrets Operator (ESO) offers a different pattern: it reads secrets from Vault and materializes them as native Kubernetes Secret objects. Workloads then consume those secrets through env, envFrom, or volume mounts—the same way they would use any other Kubernetes secret.

=== Install External Secrets Operator

Procedure

Install ESO from the official Helm chart into the external-secrets namespace:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update external-secrets
helm upgrade --install external-secrets external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true \
  --wait --timeout 10m

Confirm the operator is running:
```
oc get pods -n external-secrets
```

You should see the external-secrets deployment pod in Running state.

=== Configure ESO to sync from Vault

ESO reads the my-second-secret you created in the Vault UI and syncs it into a Kubernetes Secret named my-second-secret-sync.

In dev mode, ESO authenticates to Vault with the root token stored in a Kubernetes Secret. A SecretStore defines how to reach Vault; an ExternalSecret defines which Vault paths to copy and the name of the target Kubernetes Secret.

Procedure

Apply the token, SecretStore, and ExternalSecret:

oc apply -f ~/openshift-security-roadshow/setup/vault-lab/eso-secretstore.yaml

Verify ESO created the Kubernetes Secret:

oc get externalsecret,secretstore -n vault
oc get externalsecret sync-my-second-secret -n vault -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
oc get secret my-second-secret-sync -n vault -o jsonpath='{.data.username}{"\n"}' | base64 -d && echo

ExternalSecret Ready status is True, and the decoded username is workshop-user-2.

If status is SecretSyncedError and my-second-secret-sync was not created, confirm my-second-secret in the Vault UI includes both username and password, then inspect the error:

oc describe externalsecret sync-my-second-secret -n vault | tail -20
oc get externalsecret sync-my-second-secret -n vault -w

Press Ctrl+C when READY is True, then re-run the verification commands above.

=== Deploy a second application using the synced Secret

Deploy a second workload that reads credentials from the Kubernetes Secret ESO created—not from Vault Agent files.

Procedure

Deploy the application:

oc apply -f ~/openshift-security-roadshow/setup/vault-lab/eso-vault-test.yaml

oc get pods -n vault -l app=eso-read-vault-secret -w

Press Ctrl+C when the pod is 1/1 Ready.

Confirm the pod received the synced credentials:

ESO_POD=$(oc get pod -n vault -l app=eso-read-vault-secret -o jsonpath='{.items[0].metadata.name}')
oc exec -n vault "${ESO_POD}" -c app -- sh -c 'echo "username=${USERNAME}" && echo "password=${PASSWORD}"'

Expected: username=workshop-user-2 and password=workshop-password-2.

Perfect! You’ve deployed a second workload that reads credentials from the Kubernetes Secret ESO created—not from the Vault Agent Injector.

Output ends with [vault-lab-cleanup] Done. and no vault or external-secrets namespaces remaining.

Debrief

You consumed Vault secrets two ways—Agent Injector files under /vault/secrets/ and External Secrets Operator sync into a native Secret—so credentials are not baked into Git or images.

What breaks without this:

Static Secrets in etcd/Git/images → long-lived loot after a foothold
No rotation path → stolen credentials stay valid

Controls that matter: Vault (or equivalent) as source of truth, Kubernetes auth bound to namespace/SA, injector for file delivery, ESO when apps need native Secret mounts, and least-privilege Vault roles.

Quick facts: prefer short-lived, identity-bound access over copied passwords. Lab cleanup uses the Vault workshop scripts plus lab-cleanup.sh --module 201-06.

Cleanup

When you finish this module, run the workshop cleanup script from the bastion. It removes External Secrets Operator, all demo applications, Vault, and related namespaces. Then run the roadshow module cleanup script.

cd ~/openshift-days-ops-showroom/setup-scripts/vault-config-scripts
./cleanup-vault-lab.sh

cd ~/openshift-security-roadshow
bash setup/lab-cleanup.sh --module 201-06