Certificate Management & mTLS Basics (cert-manager Operator)

Table of Contents

Overview
Why it matters
What does it solve
Your Mission
Debrief
Cleanup

Duration: ~30 minutes

Overview

Learn how to deploy and configure cert-manager Operator for automated certificate provisioning, renewal, and management. You’ll implement mutual TLS (mTLS) between services to ensure authenticated and encrypted communication.

Install cert-manager, issue certificates automatically, and enable basic service-to-service mTLS with verifiable rotation. Then point the same Certificate object at an external signer (the cluster ACME ClusterIssuer) so a Route hostname is trusted by browsers and default curl—no -k, no lab CA import.

Why it matters

Manual certificate management is error-prone and leads to expired certificates causing outages. Cert-manager automates the entire certificate lifecycle—provisioning, renewal, and revocation—reducing operational overhead and security risks. Mutual TLS adds an extra layer of security by requiring both client and server authentication.

A self-signed issuer proves the API. It does not prove identity to the rest of the world: curl -k and “import this PEM” do not scale. The production hook is issuerRef on the Certificate: same CR, same renewal, different signer. Public ACME (Let’s Encrypt and similar) is already in OS and browser trust stores. An enterprise CA (Vault PKI, Venafi, AD CS) is the same idea for mTLS that must be recognized on every cluster and laptop.

What does it solve

Certificate expiration outages
Manual certificate management overhead
Unencrypted service communication
Man-in-the-middle attacks
Certificate provisioning complexity
Encryption without global trust (-k forever, or a PEM copied to every client)

Your Mission

Issue a service certificate with cert-manager so east-west traffic can use identity, not a copied PEM file. Then hook a north-south Route into the cluster’s existing ACME ClusterIssuer so the same automation produces a certificate the public trust store already accepts. You already put TLS on a Route in 101-12; this lab is the in-cluster certificate object plus the external-signer switch.

Do not edit or delete ClusterIssuers named selfsigned or acme-*. Do not change certificates in openshift-ingress, openshift-config, or stackrox.

Click each step only if you need a hint.

Part A: In-cluster issuance (self-signed)

This is the intro: cert-manager writes a Secret. The signer is local. Clients need --cacert or -k. That is enough for a lab mTLS mount; it is not enough for “the world trusts this name.”

Get pods in openshift-cert-manager

Confirm the operator is already running on this cluster.

oc get pods -n openshift-cert-manager

cert-manager pods should be Running. If the namespace is empty, install cert-manager Operator for Red Hat OpenShift from OperatorHub (or the openshift-cert-manager-operator Subscription) and wait for the CRD in the next step.

Get CRD certificates.cert-manager.io

Without this CRD, Certificate objects will not apply.

oc get crd certificates.cert-manager.io

The CRD exists. If not, install the operator and re-run.

List ClusterIssuers (self-signed vs ACME)

Roadshow clusters typically already have selfsigned and an ACME issuer (for example acme-bifrost-production-ddns) used for the ingress wildcard. Part A uses self-signed. Part B uses ACME. You only reference them.

oc get clusterissuer

selfsigned (or equivalent) and an acme-* issuer should show Ready=True. If selfsigned is missing, the next step creates it.

Ensure ClusterIssuer selfsigned exists

Create it only when it is absent. Do not replace an ACME issuer.

if oc get clusterissuer selfsigned >/dev/null 2>&1; then
  echo "Using existing ClusterIssuer selfsigned"
  oc get clusterissuer selfsigned
else
  oc apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
EOF
fi

Ready=True for selfsigned.

Create project 201-08-demo

Certificates and the demo Route land in this namespace.

oc new-project 201-08-demo

Now using project "201-08-demo".

Apply Certificate example-cert (in-cluster name)

secretName is where cert-manager writes tls.crt / tls.key. dnsNames must match how services will call each other. Public CAs will not sign .svc.cluster.local—that is why east-west mTLS uses a private issuer.

oc apply -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-cert
  namespace: 201-08-demo
spec:
  secretName: example-cert-tls
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
  dnsNames:
  - example.201-08-demo.svc.cluster.local
  duration: 2160h
  renewBefore: 360h
EOF

certificate.cert-manager.io/example-cert created. renewBefore is the automation: cert-manager rotates the Secret before expiry without a ticket.

Get Certificate example-cert and Secret

Wait until Ready is True, then confirm the Secret.

oc get certificate -n 201-08-demo
oc get secret example-cert-tls -n 201-08-demo

READY is True. The Secret has tls.crt / tls.key. Mount that Secret into pods for mTLS (with NetworkPolicies from 101-05 so only the peers you intend can connect).

Show the self-signed issuer (not globally trusted)

issuer equals subject (or a CA nobody’s browser has). Default curl and browsers will not trust this.

oc get secret example-cert-tls -n 201-08-demo -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -issuer -subject -dates

Issuer is not a public CA. That is expected for Part A.

Part B: External signer (ACME) — globally recognized

The hook is one field: issuerRef names a ClusterIssuer that talks to a CA outside this namespace. On this cluster that is ACME (public CA). In an enterprise it is the same CR shape pointing at Vault PKI, Venafi, Google CAS, or AD CS—so every cluster and laptop already has the root.

Signer cert-manager issuerRef Who already trusts it

Signer	cert-manager `issuerRef`	Who already trusts it
Self-signed (Part A)	`kind: ClusterIssuer`, `name: selfsigned`	Only clients you give the PEM (`--cacert` / `-k`)
Public ACME (this Part)	Existing `acme-*` ClusterIssuer — do not edit it	Browsers, OS trust stores, default `curl`. North-south Routes.
Enterprise PKI	Vault, Venafi, AD CS, cloud private CA	Org trust store / every cluster’s CA bundle. East-west mTLS that must work across clusters.

Self-signed (Part A)

kind: ClusterIssuer, name: selfsigned

Only clients you give the PEM (--cacert / -k)

Public ACME (this Part)

Existing acme-* ClusterIssuer — do not edit it

Browsers, OS trust stores, default curl. North-south Routes.

Enterprise PKI

Vault, Venafi, AD CS, cloud private CA

Org trust store / every cluster’s CA bundle. East-west mTLS that must work across clusters.

Let’s Encrypt will never issue for *.svc.cluster.local. Use ACME (or another public CA) for Route hostnames. Use a private/enterprise CA for pod-to-pod mTLS. Vault PKI wiring is 201-05; SPIFFE identity is 301-08.

Pick the Ready ACME ClusterIssuer

Do not create a new ACME account. Reuse the issuer that already signs cert-manager-ingress-cert.

ACME_ISSUER=$(oc get clusterissuer -o json | jq -r '
  .items[]
  | select(.spec.acme != null)
  | select([.status.conditions[]? | select(.type=="Ready" and .status=="True")] | length > 0)
  | .metadata.name' | head -1)
echo "ACME ClusterIssuer: ${ACME_ISSUER:-NONE}"
oc get certificate -n openshift-ingress cert-manager-ingress-cert 2>/dev/null || true

You should see an issuer name (often acme-bifrost-production-ddns). The ingress Certificate is the cluster’s already-global example. If NONE, skip the apply below and only inspect the ingress cert with openssl—your platform owner has not installed ACME.

Inspect the public ingress certificate

This leaf is already signed by a public CA. That is the “recognized globally” bar.

oc get secret cert-manager-ingress-cert -n openshift-ingress -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -issuer -subject -dates

Issuer is an ACME / Let’s Encrypt–style CA, not selfsigned. Browsers already trust it.

Deploy a small app and HTTP Route

The Route hostname is the DNS name ACME can sign. Capture it; the Certificate dnsNames must match exactly.

oc create deployment public-app -n 201-08-demo \
  --image=registry.access.redhat.com/ubi9/httpd-24
oc expose deployment public-app -n 201-08-demo --port=8080 --target-port=8080
oc rollout status deployment/public-app -n 201-08-demo --timeout=120s
oc expose service public-app -n 201-08-demo
HOST=$(oc get route public-app -n 201-08-demo -o jsonpath='{.spec.host}')
echo "$HOST"

HOST is an apps. (or similar) DNS name, not .svc.cluster.local.

Issue a Certificate from the ACME ClusterIssuer

Same Certificate kind as Part A. Only issuerRef and dnsNames change. cert-manager talks to the external ACME directory, solves the challenge already configured on that ClusterIssuer, and writes public-route-tls.

HOST=$(oc get route public-app -n 201-08-demo -o jsonpath='{.spec.host}')
ACME_ISSUER=$(oc get clusterissuer -o json | jq -r '
  .items[]
  | select(.spec.acme != null)
  | select([.status.conditions[]? | select(.type=="Ready" and .status=="True")] | length > 0)
  | .metadata.name' | head -1)
echo "Issuing for HOST=${HOST} via ${ACME_ISSUER}"
if [ -z "$ACME_ISSUER" ] || [ -z "$HOST" ]; then
  echo "Skip: need a Ready ACME ClusterIssuer and a Route host."
else
  oc apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: public-route-cert
  namespace: 201-08-demo
spec:
  secretName: public-route-tls
  issuerRef:
    name: ${ACME_ISSUER}
    kind: ClusterIssuer
  dnsNames:
  - ${HOST}
  duration: 2160h
  renewBefore: 360h
EOF
fi

certificate.cert-manager.io/public-route-cert created. Challenges can take a few minutes. If Ready stays False, oc describe certificate public-route-cert -n 201-08-demo — the ACME solver might only cover the ingress wildcard zone. You still have the global ingress cert from the previous step, and you still practiced the issuerRef hook.

Wait until public-route-cert is Ready

oc get certificate public-route-cert -n 201-08-demo
oc get secret public-route-tls -n 201-08-demo 2>/dev/null || true

READY True means the external CA signed your hostname. False + ACME errors: do not edit the ClusterIssuer; use the ingress cert as the global example and keep Part A for mTLS.

Put the ACME Secret on the Route and curl without -k

Edge TLS with the issued Secret. Default curl should verify if ACME succeeded.

HOST=$(oc get route public-app -n 201-08-demo -o jsonpath='{.spec.host}')
if oc get secret public-route-tls -n 201-08-demo >/dev/null 2>&1; then
  mkdir -p /tmp/201-08
  oc extract secret/public-route-tls -n 201-08-demo --to=/tmp/201-08 --confirm
  oc delete route public-app -n 201-08-demo
  oc create route edge public-app -n 201-08-demo --service=public-app \
    --cert=/tmp/201-08/tls.crt --key=/tmp/201-08/tls.key \
    --hostname="${HOST}" --insecure-policy=Redirect
  echo "=== issuer on the app cert ==="
  openssl x509 -in /tmp/201-08/tls.crt -noout -issuer -subject -dates
  echo "=== curl (no -k) ==="
  curl -sI "https://${HOST}" | head -15
else
  echo "No public-route-tls yet. Wildcard ingress may still be public ACME:"
  curl -sI "https://${HOST}" | head -15
fi

HTTPS 200 (or a redirect) without -k. Compare issuer= here with Part A’s self-signed issuer. If verify fails, the Route may still be on the wildcard; curl -vI shows which cert the router presented.

Part C: What “global” means for mTLS vs Routes

Compare the two leaves

echo "=== Part A self-signed (east-west) ==="
oc get secret example-cert-tls -n 201-08-demo -o jsonpath='{.data.tls\.crt}' \
  | base64 -d | openssl x509 -noout -issuer -subject 2>/dev/null || echo "missing"

echo "=== Part B ACME (north-south) ==="
if oc get secret public-route-tls -n 201-08-demo >/dev/null 2>&1; then
  oc get secret public-route-tls -n 201-08-demo -o jsonpath='{.data.tls\.crt}' \
    | base64 -d | openssl x509 -noout -issuer -subject
else
  echo "(dedicated ACME Secret not Ready — ingress wildcard is still the public example)"
  oc get secret cert-manager-ingress-cert -n openshift-ingress -o jsonpath='{.data.tls\.crt}' \
    | base64 -d | openssl x509 -noout -issuer -subject
fi

Two different issuers. Same Certificate API. The signer is what made one of them globally trusted.

Debrief

cert-manager automates issuance and renewal so expiry outages disappear; issuerRef is how you hook that automation to a CA the client already trusts. mTLS adds service-to-service identity beyond perimeter TLS.

What “recognized globally” means:

Routes / browsers — public ACME ClusterIssuer (this cluster already has one). Default trust store. No -k.
East-west mTLS — not Let’s Encrypt. An enterprise CA (or SPIFFE) whose root is in every cluster’s bundle. Same Certificate (or SPIFFE SVID), different issuer.
Automation — duration / renewBefore on the CR; you do not copy PEMs or file a ticket at 2 a.m.

What breaks without this:

Manual certs → outages at expiry and rushed key handling
Self-signed as the only issuer → encryption without identity (-k everywhere)
Cleartext or one-way TLS only → lateral sniffing and spoofing between services
Editing the shared ACME ClusterIssuer from a lab → broken ingress for everyone

ClusterIssuer/Certificate CRs, auto-renew before expiry, mTLS where trust boundaries need mutual proof, protected key material, and issuerRef aimed at a CA that matches the client trust store (public ACME vs enterprise PKI).

use public CA, private CA, or self-signed to match your trust model; renewal timing is the reliability win. Combine mTLS with 101-05 so identity and network path agree.

Cleanup

Before moving to the next module, run the lab cleanup script. It deletes project 201-08-demo and a leftover selfsigned-issuer from older lab versions. It does not delete selfsigned or ACME ClusterIssuers.

cd ~/ocp5-rhacs-showroom
bash setup/lab-cleanup.sh --module 201-08