6.1 GitOps + admission

Overview

You promote the signed app digest into stage via GitOps, then turn on admission so unsigned images cannot run. You edit one file on the stage Gitea remote. That file is admission/trust-policy.yaml (kind: TrustPolicy). A CronJob renders namespaced ImagePolicy on lw-poc-staging and lw-poc-prod. Do not apply ClusterImagePolicy. Do not use RHACS Trusted image signers for this gate.

Native ImagePolicy signedEmail must be an email. Track 5.1 identity is a Kubernetes service-account URI, so the CronJob renders policyType: PublicKey from ConfigMap lab-cosign-pubkey (the 5.3 cosign.pub you published). Keep the 5.1 service-account URI in TrustPolicy. That is still the Fulcio identity. oc tag does not copy .sig tags. Use cosign copy (and a dest-repo cosign sign --key) after the tag.

Prod GitOps is a second remote. Leave it for 6.2. app-operate NetworkPolicy on staging stays open. That is not the 4.3 build-egress lab.

Why it matters

Only the signed digest may run. Lightwell pins inside the image do not admit an unsigned tag. Unsigned deny is a pull failure (ImagePullBackOff), not oc apply stderr.

You fill TrustPolicy (enforce: true, live digest, pipeline service-account subject) and run cosign copy after oc tag because signatures do not travel with oc tag by themselves. Prod Argo stays on the prod remote.

Lightwell Network and Clearinghouse Premier are already in the image as GAVs. Admission does not inspect Maven. A Premier member version does not skip unsigned deny.

What does it solve

  • Unsigned images that still deploy

  • Signatures that are not copied after oc tag

  • Using ClusterImagePolicy instead of namespaced ImagePolicy

  • Mixing stage and prod remotes

Stage vs prod remotes

Stage versus prod Gitea remotes

Discover URLs from userinfo. The Gitea password is the value of ConfigMap key student_password (not the key name). Do not put that value in Git or a screenshot.

oc -n gitea get configmap demo-userinfo-gitea \
  -o jsonpath='stage={.data.student_gitops_repo_url}{"\n"}prod={.data.student_prod_gitops_repo_url}{"\n"}promote_ns={.data.student_promote_namespace}{"\n"}build_ns={.data.student_build_namespace}{"\n"}argo={.data.student_argocd_app}{"\n"}'
oc -n tssc-admission get configmap demo-userinfo-admission \
  -o jsonpath='scored={.data.scored_kind}{"\n"}live={.data.live_kind}{"\n"}hint={.data.unsigned_deny_hint}{"\n"}'

Worked example (not the Check)

# Worked example ONLY — wrong name, enforce still false, fake identity.
# Do not copy admission-docs example-trust-policy.yaml as the scored file.
apiVersion: tssc.workshop/v1
kind: TrustPolicy
metadata:
  name: example-not-scored
spec:
  enforce: false
  scopes:
    - quay.io/example/not-the-lab-repo
  identity:
    issuer: https://example.invalid/oidc
    subject: student@example.invalid
  digest: sha256:0000000000000000000000000000000000000000000000000000000000000000

Your change: tag, stage GitOps, TrustPolicy

  1. Tag the signed image into staging (not prod):

BUILD_NS="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_build_namespace}')"
PROMOTE_NS="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_promote_namespace}')"
oc tag "${BUILD_NS}/spring-boot-lw-poc:latest" "${PROMOTE_NS}/spring-boot-lw-poc:latest"
oc -n "${PROMOTE_NS}" get istag spring-boot-lw-poc:latest \
  -o jsonpath='{.image.metadata.name}{"\n"}'
DIGEST="$(oc -n "${PROMOTE_NS}" get istag spring-boot-lw-poc:latest \
  -o jsonpath='{.image.metadata.name}')"
SRC="image-registry.openshift-image-registry.svc:5000/${BUILD_NS}/spring-boot-lw-poc@${DIGEST}"
DST="image-registry.openshift-image-registry.svc:5000/${PROMOTE_NS}/spring-boot-lw-poc@${DIGEST}"
# oc tag copies the image, not .sig tags. Copy, then sign the dest repo with the 5.3 key.
export HOME=/tmp/lab-home
mkdir -p "${HOME}"
cosign login -u unused -p "$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  image-registry.openshift-image-registry.svc:5000
cosign copy --allow-http-registry --allow-insecure-registry "${SRC}" "${DST}"
COSIGN_PASSWORD="" cosign sign --key /home/lab-user/lab-trust/cosign.key \
  --tlog-upload=false --allow-http-registry --allow-insecure-registry \
  "${DST}"

Keep the sha256: prefix. That digest must match Track 5.1.

  1. Clone the stage GitOps remote (student_gitops_repo_url). Set image.digest to that digest and replicas: 1. Commit and push. Do not edit the prod remote. Follow PROMOTE.md in that repo if you need the field names. A fresh clone does not inherit git config from the app repo — set author locally (same identity as 3.2):

STAGE_GITOPS="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_gitops_repo_url}')"
: "${STUDENT_USER:=$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_username}')}"
: "${STUDENT_PASS:=$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_password}')}"
rm -rf /tmp/gitops-stage
hostpath="${STAGE_GITOPS#https://}"; hostpath="${hostpath#http://}"
git clone "https://${STUDENT_USER}:${STUDENT_PASS}@${hostpath}" /tmp/gitops-stage
cd /tmp/gitops-stage
git config user.name student
git config user.email student@workshop.local

  1. After you push image.digest + replicas: 1, hard-refresh Argo. Automated sync often stays on the previous commit (replicas: 0 still looks Healthy). Wait until the Deployment has replicas: 1 and a Ready pod:

ARGO="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_argocd_app}')"
PROMOTE_NS="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_promote_namespace}')"
oc -n openshift-gitops annotate applications.argoproj.io "${ARGO}" \
  argocd.argoproj.io/refresh=hard --overwrite
oc -n openshift-gitops get applications.argoproj.io "${ARGO}" \
  -o jsonpath='sync={.status.sync.status} health={.status.health.status} rev={.status.sync.revision}{"\n"}'
oc -n "${PROMOTE_NS}" get deploy spring-boot-lw-poc \
  -o jsonpath='replicas={.spec.replicas} ready={.status.readyReplicas}{"\n"}'
Argo Application lw-poc-staging Healthy and Synced after 6.1

  1. Edit admission/trust-policy.yaml on the same stage remote. Do not change apiVersion / kind. Set enforce: true. Replace every REPLACE_ME_* with live values: issuer https://kubernetes.default.svc (or userinfo oidc_issuer_hint), subject the pipeline SA URI from 5.1 (not an invented email), digest the signed sha256:…, scopes the internal ImageStream repos staging and prod actually pull — discover with oc get is. Confirm ConfigMap tssc-admission/lab-cosign-pubkey still has your 5.3 PEM. Do not paste Fulcio / Rekor PEMs (the CronJob injects those). The CronJob sets signedIdentity.matchPolicy: ExactRepository to the build ImageStream so dest pulls accept the signature identity from lw-poc-build.

Learner TrustPolicy file versus CronJob-rendered ImagePolicy on stage and prod

CronJob trust-policy-apply in tssc-admission polls about every two minutes. Pass when ImagePolicy exists in both lw-poc-staging and lw-poc-prod (name from userinfo policy_name).

Check: unsigned deny; stage Healthy

Native ImagePolicy is CRI-O policy.json: oc apply of an unsigned Deployment can succeed; the pod fails at pull. Kyverno fallback (if the claim uses it) denies at admission instead — still look at the pod / events.

The gate only covers spec.scopes. Tag an unsigned digest onto the same ImageStream and deploy that tag. Do not point GitOps image.digest at it (that would flap the Healthy Check).

BUILD_NS="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_build_namespace}')"
PROMOTE_NS="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_promote_namespace}')"
ARGO="$(oc -n gitea get configmap demo-userinfo-gitea -o jsonpath='{.data.student_argocd_app}')"
# Unsigned digest, in-scope name:
oc tag "${BUILD_NS}/openjdk-21-builder:latest" "${PROMOTE_NS}/spring-boot-lw-poc:unsigned"
oc -n "${PROMOTE_NS}" create deployment unsigned-deny-test \
  --image="image-registry.openshift-image-registry.svc:5000/${PROMOTE_NS}/spring-boot-lw-poc:unsigned"
oc -n "${PROMOTE_NS}" rollout status deploy/unsigned-deny-test --timeout=30s || true
oc -n "${PROMOTE_NS}" get pods -l app=unsigned-deny-test
oc -n "${PROMOTE_NS}" get events --field-selector reason=Failed --sort-by=.lastTimestamp | tail -10
oc -n "${PROMOTE_NS}" delete deployment unsigned-deny-test
oc -n openshift-gitops get applications.argoproj.io "${ARGO}" \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status
oc -n "${PROMOTE_NS}" get deploy spring-boot-lw-poc \
  -o jsonpath='replicas={.spec.replicas} ready={.status.readyReplicas}{"\n"}'
oc -n "${PROMOTE_NS}" get imagepolicy
oc -n lw-poc-prod get imagepolicy

Pass when: TrustPolicy has no REPLACE_ME and enforce: true; ImagePolicy is live on staging and prod; the :unsigned pod is ImagePullBackOff (or signature error); lw-poc-staging stays Healthy on the signed digest. Do not retarget prod Argo at the stage remote.

Throwaway unsigned tag pod in ImagePullBackOff while signed latest stays running
Fill the report

The Job also grades ConfigMap report-15-gitops-admission key live_api. Set it to imagepolicy. Empty or REPLACE_ME fails even if cluster state is correct. Do not copy validate-docs.

oc -n lw-poc-validate edit configmap report-15-gitops-admission

The scored Check is Job validate-15-gitops-admission in lw-poc-validate. Direct nav URLs still work (honor system). Unlimited retries. There is no Solve.

Run the Validate Job
oc -n lw-poc-validate delete job validate-15-gitops-admission --ignore-not-found
TPL='job-15.yaml'
oc -n lw-poc-validate get configmap validate-job-templates \
  -o go-template="{{ index .data \"${TPL}\" }}" | oc create -f -
oc -n lw-poc-validate logs -f job/validate-15-gitops-admission

Expect CHECK PASSED: before you continue. CHECK FAILED: names the object to fix. Delete the Job and create it again as often as you need.

Continue to 6.2 when you are ready.

Official documentation

Callouts (not Checks)

  • A production VM is the same signed digest on bootc / Satellite. No VM in this lab.

  • Canary and blue-green are other strategies for that digest. This lab uses rolling (default Deployment).

Key takeaways

  • Admission is namespaced ImagePolicy from one TrustPolicy file you own. It is not ClusterImagePolicy.

  • Lightwell Network and Clearinghouse Premier GAVs do not bypass unsigned deny.

  • Service-account URI subjects render PublicKey from 5.3 lab-cosign-pubkey. oc tag needs cosign copy.

  • Promotion to prod is a git commit to a second remote (6.2).

cd ~/ocp5-rhacs-showroom
bash setup/lab-cleanup.sh --module tssc-15