Module 07: GitOps vulnerability management
This module is a presenter-led walkthrough of how RHACS 5.0 answers a GitOps customer’s six vulnerability-management requirements. The shop-api and shop-web workloads run across demo-dev, demo-stage, demo-prod, and demo-platform. Images come from quay.io/mfoster. Setup is applied by basic-setup/10-deploy-vulnmgmt-demo.sh.
Estimated time: 20-25 minutes (Part 5 includes an optional branch-environment step)
|
Unique walkthrough: This module does not replace Module 02. Module 02 remains the general Results, image-age, and risk tour. This module is a separate GitOps shop, Quay, and Argo CD requirements walkthrough (R1 through R6). |
The customer’s one question: Which vulnerabilities actually affect the software I have in use?
| ID | Requirement | What we show |
|---|---|---|
R1 |
Version and stage tracking |
Namespaces, labels, collections, watched images, registry copies that keep the same digest |
R2 |
Persistent, versioned triage |
OpenVEX in Git, exception API, requester versus approver |
R3 |
Layer attribution |
Base image versus application layer, policy as code |
R4 |
Scanner hinting |
Red Hat VEX and content manifests in Scanner V4 |
R5 |
Everything API-driven |
|
R6 |
Dev teams as primary users |
Scoped RBAC, OpenShift console Security tab |
Before any CLI step, source the environment in the terminal:
source ~/.bashrc
source ~/rhacs-demo/basic-setup/vulnmgmt/env.sh
Part 1: Who owns this CVE
Know
Context: Platform teams ship base images (OpenJDK, JBoss EAP, Kafka). Application teams add their own layers. A flat CVE list makes every finding look like the developer’s problem.
Why layer attribution matters:
-
The same library can appear twice in one image: a Red Hat-patched copy in the base, and an unpatched copy the developer copied in
-
Without layer context, teams re-triage Log4Shell against a version that is already patched
-
Policies should be able to fail a build for an application-layer CVE without treating every base-image finding as the app team’s defect
What RHACS 5.0 provides:
-
Image scan data includes component version, file location, and layer index
-
After base images are registered, RHACS can classify layers as base versus application (this can lag by one re-scan)
-
SecurityPolicy custom resources live in Git. The UI shows them as managed by Kubernetes
-
roxctl image checkis the CI gate.FAIL_BUILD_ENFORCEMENTfails the pipeline
Tech Preview (policy Filters wizard):
ROX_POLICY_FILTERS_UI adds a Filters step in the create-policy wizard: all layers, skip base layers, or skip application layers. That evaluation filter is UI-only in 5.0. It is not in SecurityPolicy CRs yet.
Show
What I say:
"This shop-api image has two copies of log4j-core. One is in the base image the platform team owns. The other is a JAR the application team copied in. Same library, two owners, two risk profiles. That is the split this customer asked for."
What I do:
-
From the left side menu, expand Vulnerability Management and click Results.
-
Filter to namespace
demo-dev, then open the imageshop-api:1.0.0. -
In the component or vulnerability list, find
log4j-core. Point out that two versions appear. One is the developer’s copy under aCOPYpath such asdeployments/lib/. The other is a.redhat-build in a base-image path. Do not quote layer numbers from a slide. Live values come from the API next. -
Show the same split from the API:
IMAGE_ID=$(bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/resolve-image-id.sh shop-api:1.0.0) echo "IMAGE_ID=${IMAGE_ID}" rox "https://${CENTRAL_ROUTE}/v1/images/${IMAGE_ID}" | \ jq '[((.scan.components) // [])[] | select(.name | tostring | test("log4j-core")) | { version, layerIndex, location, vuln_count: ((.vulns // []) | length), vulns: [(.vulns // [])[] | {cve, severity: .severity, fixedBy}] }]'"The application copy carries Log4Shell. The Red Hat-patched copy in the base does not. File path plus layer index tells you which Containerfile instruction introduced each one."
-
Optional: show the top Dockerfile instructions from image metadata:
rox "https://${CENTRAL_ROUTE}/v1/images/${IMAGE_ID}" | \ jq '[((.metadata.v1.layers) // [])[-8:][] | {instruction, value}]' -
Open the Log4Shell policy as code:
cat ~/rhacs-demo/basic-setup/vulnmgmt/acs/policies/demo-log4shell-block.yamlPoint to
enforcementActions: FAIL_BUILD_ENFORCEMENT, scopedemo-.*, and bothCVE-2021-44228andCVE-2021-45046(Scanner V4 may report the follow-up CVE). -
In the RHACS console, go to Platform Configuration > Policy Management. Search for
Demo:. Open Demo: Log4Shell CVE-2021-44228. Point to the Managed by: Kubernetes badge. It is read-only in the UI. -
Run the build-time gate.
shop-api:1.0.0should fail Log4Shell.shop-api:1.1.0should not:roxctl image check --endpoint "${ROX_ENDPOINT}" \ --token-file <(echo "$ROX_API_TOKEN") \ --image "${REG}/shop-api:1.0.0" --insecure-skip-tls-verify \ -o json 2>/dev/null | jq '{ summary, policies: [.results[].violatedPolicies[] | { policy: .name, severity, breaks_build: .failingCheck, violations: [.violation[]?][:3] }] }' || true roxctl image check --endpoint "${ROX_ENDPOINT}" \ --token-file <(echo "$ROX_API_TOKEN") \ --image "${REG}/shop-api:1.1.0" --insecure-skip-tls-verify \ -o json 2>/dev/null | jq '{ summary, policies: [.results[].violatedPolicies[] | { policy: .name, severity, breaks_build: .failingCheck }] }' || true -
Optional Filters wizard (Tech Preview): Platform Configuration > Policy Management > Create Policy. After criteria and scope, the Filters step offers skip-base versus skip-application layers. Call this a preview. It is not GitOps-manageable in 5.0.
What they should notice:
-
Two log4j-core copies, two owners, two CVE sets
-
Policies are YAML in Git and read-only in the UI
-
CI uses
roxctl image check. No one clicks Fail in the console to block a build
|
Presenter tip: Component Layer Type ( |
Part 2: False positive reduction with Red Hat VEX
Know
Context: Quay Clair matched JARs to upstream library versions. Red Hat had already backported fixes inside the product, so the Security tab filled with findings that were not actionable.
How Scanner V4 reduces that noise:
-
Official Red Hat images include content manifests (typically under
/root/buildinfo/content_manifests/) -
Scanner V4 maps those components to Red Hat product CPEs and uses Red Hat vulnerability sources
-
The flag
ROX_SCANNER_V4_RED_HAT_LAYERS_RED_HAT_VULNS_ONLY(enabled in this lab) uses only Red Hat sources for packages in official Red Hat layers -
Since RHACS 4.11.1, Scanner V4 also honors
not_vulnerableassertions from Red Hat CSAF/VEX -
Copy the same JARs into a plain UBI layer and the manifests are gone. The scanner falls back to upstream matching
This VEX path applies to Red Hat-shipped content. Third-party JARs still use the exception API in Part 3.
Show
What I say:
"This is the Clair problem in numbers. Same JARs, two images. The official Red Hat image keeps content manifests. The repacked image does not. Watch how many Java CVEs appear when the scanner can no longer tell it is looking at Red Hat content."
What I do:
-
Run the comparison (live totals vary; treat any slide that says 37 versus 540 as ballpark only):
bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/compare-eap.sh -
Point to total vulnerabilities and the Java CVE counts. Official should be near zero Java CVEs. Repacked should show a large Java CVE set.
-
Compare one component, for example jackson-databind:
echo "Official jackson-databind CVEs:" jq '[.result.vulnerabilities[] | select(.componentName=="com.fasterxml.jackson.core:jackson-databind")] | length' \ ~/rhacs-demo/basic-setup/vulnmgmt/output/eap8-official.json echo "Repacked jackson-databind CVEs:" jq '[.result.vulnerabilities[] | select(.componentName=="com.fasterxml.jackson.core:jackson-databind")] | length' \ ~/rhacs-demo/basic-setup/vulnmgmt/output/eap8-repacked.json -
Restate the design rule: build base images
FROMofficial Red Hat images. Do not copy JARs out into a new layer if you want Red Hat VEX to apply.
What they should notice:
-
Identical JAR content, very different CVE counts
-
The flag is not on by default. This lab turns it on in script 08
-
Official-image leftovers are typically OS RPMs with Red Hat advisories, not Java false positives
|
Presenter tip: If the two scans are still running or the images are not in Quay yet, skip the numbers and keep the mechanism: manifests plus VEX versus upstream matching. Do not invent counts. |
Part 3: Persistent triage with OpenVEX
Know
Context: False positives and deferrals must not be re-typed after every rebuild. The customer wants an open standard in Git, plus an API the pipeline can call.
What this lab implements:
-
shop.openvex.jsonis the source of truth -
not_affectedbecomes a false-positive exception -
affectedwith a calendar date in the action statement becomes a deferral with TIME expiry (customDateis enabled in script 05) -
Scope is registry
quay.io, remotemfoster/shop-api, tag.*. Exceptions follow new tags of that image name, not a single digest -
In 5.0, an exception on
shop-apidoes not leak toprod-mirror-shop-apieven when the digest matches. Extend scope explicitly if prod should inherit the decision -
demo-team-a-devcan request.demo-vuln-approvercan approve. Requester tokens cannot call approve
RHACS does not ingest OpenVEX natively. The bridge is a short script the pipeline owns.
Show
What I say:
"Git holds the assessment. The pipeline translates OpenVEX into RHACS exceptions. Approval is a second identity. Rebuilds do not wipe the decision, because we scoped by image name regex, not digest."
What I do:
-
Show the VEX document:
cat ~/rhacs-demo/basic-setup/vulnmgmt/vex/shop.openvex.json"CVE-2022-42889 is not affected: StringSubstitutor is not on the execute path. CVE-2021-44228 is affected, with a planned fix date of 2026-10-01."
-
Glance at the bridge (create plus optional approve):
sed -n '1,80p' ~/rhacs-demo/basic-setup/vulnmgmt/scripts/vex2acs.sh -
List exceptions already created by setup:
rox "https://${CENTRAL_ROUTE}/v2/vulnerability-exceptions" | \ jq '[.exceptions[] | select(.cves[]? == "CVE-2022-42889" or .cves[]? == "CVE-2021-44228") | { name, targetState, status, cves, scope: .scope.imageScope }]'Expect an approved false positive for CVE-2022-42889 and an approved deferral for CVE-2021-44228, both scoped to
mfoster/shop-apiwith tag.*. -
Explain the RBAC split. A requester token cannot approve:
echo "Requester role cannot approve. Expect not authorized:" curl -sk -o /tmp/vex-approve-denied.json -w "HTTP %{http_code}\n" \ -H "Authorization: Bearer ${TEAM_A_TOKEN}" \ -H "Content-Type: application/json" \ -X POST "https://${CENTRAL_ROUTE}/v2/vulnerability-exceptions/does-not-exist/approve" \ -d '{"comment":"demo"}' jq -r '.message // .' /tmp/vex-approve-denied.json -
In Vulnerability Management > Exception Management (or the deferred filter on Results), show that CVE-2021-44228 can be hidden as deferred while the exception is in force.
What they should notice:
-
OpenVEX in Git, exceptions in RHACS, approval as a separate audit step
-
Tag regex
.*covers future shop-api tags -
Prod-mirror is a different image name. Copying the digest does not copy the exception
|
Presenter tip: If Exception Management is empty, re-run |
Part 4: Stages, CI, and CVE impact
Know
Context: One application, many component images, three stages, and registry copies between them. The customer still needs to know which version is where, and which running software a given CVE hits.
R1 in this lab:
-
Stages are namespaces:
demo-dev,demo-stage,demo-prod, labeledstage=andteam= -
Prod pulls
quay.io/mfoster/prod-mirror-shop-api:1.0.0(flat name, notprod-mirror/shop-api) -
RHACS 5.0 tracks name plus digest. Same digest, two names, two records
-
Collections:
shop-all,shop-prod,team-a-everything,platform-base-images -
Watched images cover artifacts that are built but not deployed yet
-
There is no first-class application-version object. Labels such as
app.version=1.0.0belong to the customer’s model
R5 in this lab:
-
roxctl image check,image scan,image sbom,deployment check -
REST for collections, exceptions, base images, watched images, RBAC
-
Policies as SecurityPolicy CRs
The umbrella question (D8): given a CVE ID, which deployments, which stages, which layer, is it fixable, and is it already deferred?
Show
What I say:
"RHACS is not your CMDB for application versions. It is the image-level truth. Your GitOps labels and collections answer 'what is in prod.' One CVE query then answers who is hit, in which stage, and who owns the layer."
What I do:
-
Show the three stages and prod-mirror image name:
oc get pods -n demo-dev -l app=shop oc get pods -n demo-stage -l app=shop oc get pods -n demo-prod -l app=shop oc get deploy shop-api -n demo-prod -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' -
Same digest, two names:
rox -G --data-urlencode 'query=Image:r/.*shop-api' --data-urlencode 'pagination.limit=50' \ "https://${CENTRAL_ROUTE}/v1/images" | \ jq '[.images[]? | {name: (.name.fullName // .name), id}]'"Prod uses
prod-mirror-shop-api. That is a different image name with the same digest asshop-api:1.0.0." -
Collections and deployment labels:
rox "https://${CENTRAL_ROUTE}/v1/collections" | \ jq '[.collections[] | select(.name | test("shop-|team-a-|platform-base")) | {name, description}]' oc get deployments -n demo-dev -l app=shop --show-labels -
Watched images (bases from setup). Optionally watch the 1.1.0 release candidate:
rox "https://${CENTRAL_ROUTE}/v1/watchedimages" | jq '.watchedImages' rox "https://${CENTRAL_ROUTE}/v1/watchedimages" -X POST \ -H 'Content-Type: application/json' \ -d '{"name":"'"${REG}"'/shop-api:1.1.0"}' -
Scan and SBOM for pipeline consumption (counts are live, not a script):
roxctl image scan --endpoint "${ROX_ENDPOINT}" \ --token-file <(echo "$ROX_API_TOKEN") \ --image "${REG}/shop-api:1.0.0" --force --insecure-skip-tls-verify \ -o json | jq '.result.summary' roxctl image sbom --endpoint "${ROX_ENDPOINT}" \ --token-file <(echo "$ROX_API_TOKEN") \ --image "${REG}/shop-api:1.0.0" --insecure-skip-tls-verify \ | jq '{spdxVersion, packages_count: (.packages | length)}' -
Deployment pre-check as an Argo CD gate:
roxctl deployment check --endpoint "${ROX_ENDPOINT}" \ --token-file <(echo "$ROX_API_TOKEN") \ -f ~/demo-applications/vulnmgmt/demo-dev/shop-api.yaml \ --insecure-skip-tls-verify -o json 2>/dev/null | jq '{ summary, policies: [.results[].violatedPolicies[]? | {policy: .name, severity, breaks_build: .failingCheck}][:8] }' || trueIf that yaml is not under
~/demo-applications/vulnmgmt, use the sibling clone next torhacs-demo. -
Answer "which running software is affected by CVE-2021-44228?":
bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/cve-impact.sh CVE-2021-44228 bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/cve-impact.sh CVE-2021-44228 --team team-a rox "https://${CENTRAL_ROUTE}/v2/vulnerability-exceptions" | \ jq '.exceptions[] | select(.cves[]? == "CVE-2021-44228") | {status, targetState, expiry: .deferralRequest.expiry}'"Expect shop-api in demo-dev, demo-stage, and demo-prod. layerIndex on the vulnerable log4j-core is the application COPY, not the platform base. The deferral from Part 3 is still APPROVED."
What they should notice:
-
Stages are namespaces and labels, not a built-in RHACS version object
-
prod-mirror-shop-apiis a sibling repository name, not a nested path -
One CVE query plus exceptions plus layer index is the customer’s daily workflow
-
Every step is scriptable. The human step is the assessment stored in OpenVEX
Part 5: Dev teams and branch environments
Know
Context: Developers should not live in the Central UI. They need their namespaces, a request path for exceptions, and vulnerabilities in the OpenShift console they already use.
R6 in this lab:
-
Access scope
team-a-namespacesmatchesteam=team-a -
Role
demo-team-a-devuses the Vulnerability Management Requester permission set, scoped to that access scope -
Role
demo-vuln-approveruses the Approver permission set with unrestricted scope -
The OpenShift console Security tab is the ACS plugin on the deployment
-
Reports can use the
team-a-everythingcollection
Optional D9: Branch CI creates short-lived namespaces. Sensor sees them without onboarding. After delete, the deployment is gone. The image can remain searchable if it was watched or still within retention.
Show
What I say:
"Dev teams stay in OpenShift. They see their own namespaces. They can request a deferral. They cannot approve it. Branch environments appear when Argo CD creates them and disappear when the namespace is deleted."
What I do:
-
Show the demo roles:
rox "https://${CENTRAL_ROUTE}/v1/roles" | \ jq '[.roles[] | select(.name | startswith("demo-")) | {name, permissionSetId, accessScopeId}]' -
Show scoped visibility with the team-a token (platform workloads should be absent):
curl -sk -H "Authorization: Bearer ${TEAM_A_TOKEN}" \ "https://${CENTRAL_ROUTE}/v1/deployments" | \ jq '[.deployments[] | {name, namespace}]' -
Open the OpenShift console: {openshift_cluster_console_url}[window=blank]
Navigate to Workloads > Deployments, project
demo-dev, deploymentshop-api, then the Security tab."The developer never had to open Central to see these CVEs."
-
Optional: Vulnerability Management > Vulnerability Reporting. Create a report on collection
team-a-everything. The CSV is team-a only. -
Optional branch environment (skip if short on time):
oc create namespace demo-branch-feature-x --dry-run=client -o yaml | oc apply -f - oc label namespace demo-branch-feature-x demo=vulnmgmt stage=branch team=team-a --overwrite cat <<EOF | oc apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: shop-api namespace: demo-branch-feature-x labels: {app: shop, component: api, app.version: "1.0.0-feature-x", team: team-a} spec: replicas: 1 selector: {matchLabels: {app: shop, component: api}} template: metadata: labels: {app: shop, component: api, app.version: "1.0.0-feature-x", team: team-a} spec: containers: - name: api image: ${REG}/shop-api:feature-x-abc1234 command: ["sleep", "infinity"] securityContext: allowPrivilegeEscalation: false runAsNonRoot: true capabilities: {drop: ["ALL"]} seccompProfile: {type: RuntimeDefault} EOFWait until the deployment is listed in Results.
team-a-everythingincludes it because ofteam=team-a. Then delete:oc delete namespace demo-branch-feature-x rox -G --data-urlencode 'query=Image Tag:r/feature-.*' --data-urlencode 'pagination.limit=50' \ "https://${CENTRAL_ROUTE}/v1/images" | \ jq '[.images[]? | (.name.fullName // .name)]'
What they should notice:
-
Scoped tokens hide
demo-platform -
Console Security tab is the daily view
-
Branch namespaces need no manual RHACS onboarding
|
Presenter tip: If |
Requirements coverage and known gaps
| Req | Capability | Parts |
|---|---|---|
R1 |
Namespaces, labels, collections, watched images, prod-mirror names |
4, 5 |
R2 |
Exception API and OpenVEX bridge |
3 |
R3 |
Base versus application layer, policy as code |
1 |
R4 |
Red Hat VEX and content manifests |
2 |
R5 |
roxctl, REST, SecurityPolicy CRs |
1, 4 |
R6 |
Access scopes, console plugin, scoped reports |
5 |
Gaps to say out loud:
-
No native OpenVEX ingest (bridge script is the customer’s to own)
-
Layer-type policy filters are Tech Preview and UI-only. Not in CRs yet
-
No first-class application version entity. The customer owns that model
-
Red Hat VEX filtering applies to Red Hat-shipped content. Third-party JARs use exceptions
-
ROX_SCANNER_V4_RED_HAT_LAYERS_RED_HAT_VULNS_ONLYis not enabled by default