Module 07: GitOps vulnerability management

This module is a presenter-led walkthrough of how RHACS 5.0 answers a GitOps customer’s six vulnerability-management requirements. The shop-api and shop-web workloads run across demo-dev, demo-stage, demo-prod, and demo-platform. Images come from quay.io/mfoster. Setup is applied by basic-setup/10-deploy-vulnmgmt-demo.sh.

Estimated time: 20-25 minutes (Part 5 includes an optional branch-environment step)

Unique walkthrough: This module does not replace Module 02. Module 02 remains the general Results, image-age, and risk tour. This module is a separate GitOps shop, Quay, and Argo CD requirements walkthrough (R1 through R6).

The customer’s one question: Which vulnerabilities actually affect the software I have in use?

ID Requirement What we show

R1

Version and stage tracking

Namespaces, labels, collections, watched images, registry copies that keep the same digest

R2

Persistent, versioned triage

OpenVEX in Git, exception API, requester versus approver

R3

Layer attribution

Base image versus application layer, policy as code

R4

Scanner hinting

Red Hat VEX and content manifests in Scanner V4

R5

Everything API-driven

roxctl, REST, SecurityPolicy CRs

R6

Dev teams as primary users

Scoped RBAC, OpenShift console Security tab

Before any CLI step, source the environment in the terminal:

source ~/.bashrc
source ~/rhacs-demo/basic-setup/vulnmgmt/env.sh

Part 1: Who owns this CVE

Know

Context: Platform teams ship base images (OpenJDK, JBoss EAP, Kafka). Application teams add their own layers. A flat CVE list makes every finding look like the developer’s problem.

Why layer attribution matters:

  • The same library can appear twice in one image: a Red Hat-patched copy in the base, and an unpatched copy the developer copied in

  • Without layer context, teams re-triage Log4Shell against a version that is already patched

  • Policies should be able to fail a build for an application-layer CVE without treating every base-image finding as the app team’s defect

What RHACS 5.0 provides:

  • Image scan data includes component version, file location, and layer index

  • After base images are registered, RHACS can classify layers as base versus application (this can lag by one re-scan)

  • SecurityPolicy custom resources live in Git. The UI shows them as managed by Kubernetes

  • roxctl image check is the CI gate. FAIL_BUILD_ENFORCEMENT fails the pipeline

Tech Preview (policy Filters wizard):

ROX_POLICY_FILTERS_UI adds a Filters step in the create-policy wizard: all layers, skip base layers, or skip application layers. That evaluation filter is UI-only in 5.0. It is not in SecurityPolicy CRs yet.

Show

What I say:

"This shop-api image has two copies of log4j-core. One is in the base image the platform team owns. The other is a JAR the application team copied in. Same library, two owners, two risk profiles. That is the split this customer asked for."

What I do:

  1. From the left side menu, expand Vulnerability Management and click Results.

  2. Filter to namespace demo-dev, then open the image shop-api:1.0.0.

  3. In the component or vulnerability list, find log4j-core. Point out that two versions appear. One is the developer’s copy under a COPY path such as deployments/lib/. The other is a .redhat- build in a base-image path. Do not quote layer numbers from a slide. Live values come from the API next.

  4. Show the same split from the API:

    IMAGE_ID=$(bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/resolve-image-id.sh shop-api:1.0.0)
    echo "IMAGE_ID=${IMAGE_ID}"
    rox "https://${CENTRAL_ROUTE}/v1/images/${IMAGE_ID}" | \
      jq '[((.scan.components) // [])[] | select(.name | tostring | test("log4j-core")) | {
        version,
        layerIndex,
        location,
        vuln_count: ((.vulns // []) | length),
        vulns: [(.vulns // [])[] | {cve, severity: .severity, fixedBy}]
      }]'

    "The application copy carries Log4Shell. The Red Hat-patched copy in the base does not. File path plus layer index tells you which Containerfile instruction introduced each one."

  5. Optional: show the top Dockerfile instructions from image metadata:

    rox "https://${CENTRAL_ROUTE}/v1/images/${IMAGE_ID}" | \
      jq '[((.metadata.v1.layers) // [])[-8:][] | {instruction, value}]'
  6. Open the Log4Shell policy as code:

    cat ~/rhacs-demo/basic-setup/vulnmgmt/acs/policies/demo-log4shell-block.yaml

    Point to enforcementActions: FAIL_BUILD_ENFORCEMENT, scope demo-.*, and both CVE-2021-44228 and CVE-2021-45046 (Scanner V4 may report the follow-up CVE).

  7. In the RHACS console, go to Platform Configuration > Policy Management. Search for Demo:. Open Demo: Log4Shell CVE-2021-44228. Point to the Managed by: Kubernetes badge. It is read-only in the UI.

  8. Run the build-time gate. shop-api:1.0.0 should fail Log4Shell. shop-api:1.1.0 should not:

    roxctl image check --endpoint "${ROX_ENDPOINT}" \
      --token-file <(echo "$ROX_API_TOKEN") \
      --image "${REG}/shop-api:1.0.0" --insecure-skip-tls-verify \
      -o json 2>/dev/null | jq '{
        summary,
        policies: [.results[].violatedPolicies[] | {
          policy: .name, severity, breaks_build: .failingCheck,
          violations: [.violation[]?][:3]
        }]
      }' || true
    
    roxctl image check --endpoint "${ROX_ENDPOINT}" \
      --token-file <(echo "$ROX_API_TOKEN") \
      --image "${REG}/shop-api:1.1.0" --insecure-skip-tls-verify \
      -o json 2>/dev/null | jq '{
        summary,
        policies: [.results[].violatedPolicies[] | {
          policy: .name, severity, breaks_build: .failingCheck
        }]
      }' || true
  9. Optional Filters wizard (Tech Preview): Platform Configuration > Policy Management > Create Policy. After criteria and scope, the Filters step offers skip-base versus skip-application layers. Call this a preview. It is not GitOps-manageable in 5.0.

What they should notice:

  • Two log4j-core copies, two owners, two CVE sets

  • Policies are YAML in Git and read-only in the UI

  • CI uses roxctl image check. No one clicks Fail in the console to block a build

Presenter tip: Component Layer Type (APPLICATION / BASE_IMAGE) can lag until the next scan after base-image registration. Layer index and file path are enough for the live demo. resolve-image-id.sh looks up shop-api:1.0.0 by image name and scans it if Sensor has not reported the deployment yet. Do not query /v1/images/null.

Part 2: False positive reduction with Red Hat VEX

Know

Context: Quay Clair matched JARs to upstream library versions. Red Hat had already backported fixes inside the product, so the Security tab filled with findings that were not actionable.

How Scanner V4 reduces that noise:

  • Official Red Hat images include content manifests (typically under /root/buildinfo/content_manifests/)

  • Scanner V4 maps those components to Red Hat product CPEs and uses Red Hat vulnerability sources

  • The flag ROX_SCANNER_V4_RED_HAT_LAYERS_RED_HAT_VULNS_ONLY (enabled in this lab) uses only Red Hat sources for packages in official Red Hat layers

  • Since RHACS 4.11.1, Scanner V4 also honors not_vulnerable assertions from Red Hat CSAF/VEX

  • Copy the same JARs into a plain UBI layer and the manifests are gone. The scanner falls back to upstream matching

This VEX path applies to Red Hat-shipped content. Third-party JARs still use the exception API in Part 3.

Show

What I say:

"This is the Clair problem in numbers. Same JARs, two images. The official Red Hat image keeps content manifests. The repacked image does not. Watch how many Java CVEs appear when the scanner can no longer tell it is looking at Red Hat content."

What I do:

  1. Run the comparison (live totals vary; treat any slide that says 37 versus 540 as ballpark only):

    bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/compare-eap.sh
  2. Point to total vulnerabilities and the Java CVE counts. Official should be near zero Java CVEs. Repacked should show a large Java CVE set.

  3. Compare one component, for example jackson-databind:

    echo "Official jackson-databind CVEs:"
    jq '[.result.vulnerabilities[] | select(.componentName=="com.fasterxml.jackson.core:jackson-databind")] | length' \
      ~/rhacs-demo/basic-setup/vulnmgmt/output/eap8-official.json
    echo "Repacked jackson-databind CVEs:"
    jq '[.result.vulnerabilities[] | select(.componentName=="com.fasterxml.jackson.core:jackson-databind")] | length' \
      ~/rhacs-demo/basic-setup/vulnmgmt/output/eap8-repacked.json
  4. Restate the design rule: build base images FROM official Red Hat images. Do not copy JARs out into a new layer if you want Red Hat VEX to apply.

What they should notice:

  • Identical JAR content, very different CVE counts

  • The flag is not on by default. This lab turns it on in script 08

  • Official-image leftovers are typically OS RPMs with Red Hat advisories, not Java false positives

Presenter tip: If the two scans are still running or the images are not in Quay yet, skip the numbers and keep the mechanism: manifests plus VEX versus upstream matching. Do not invent counts.

Part 3: Persistent triage with OpenVEX

Know

Context: False positives and deferrals must not be re-typed after every rebuild. The customer wants an open standard in Git, plus an API the pipeline can call.

What this lab implements:

  • shop.openvex.json is the source of truth

  • not_affected becomes a false-positive exception

  • affected with a calendar date in the action statement becomes a deferral with TIME expiry (customDate is enabled in script 05)

  • Scope is registry quay.io, remote mfoster/shop-api, tag .*. Exceptions follow new tags of that image name, not a single digest

  • In 5.0, an exception on shop-api does not leak to prod-mirror-shop-api even when the digest matches. Extend scope explicitly if prod should inherit the decision

  • demo-team-a-dev can request. demo-vuln-approver can approve. Requester tokens cannot call approve

RHACS does not ingest OpenVEX natively. The bridge is a short script the pipeline owns.

Show

What I say:

"Git holds the assessment. The pipeline translates OpenVEX into RHACS exceptions. Approval is a second identity. Rebuilds do not wipe the decision, because we scoped by image name regex, not digest."

What I do:

  1. Show the VEX document:

    cat ~/rhacs-demo/basic-setup/vulnmgmt/vex/shop.openvex.json

    "CVE-2022-42889 is not affected: StringSubstitutor is not on the execute path. CVE-2021-44228 is affected, with a planned fix date of 2026-10-01."

  2. Glance at the bridge (create plus optional approve):

    sed -n '1,80p' ~/rhacs-demo/basic-setup/vulnmgmt/scripts/vex2acs.sh
  3. List exceptions already created by setup:

    rox "https://${CENTRAL_ROUTE}/v2/vulnerability-exceptions" | \
      jq '[.exceptions[] | select(.cves[]? == "CVE-2022-42889" or .cves[]? == "CVE-2021-44228") | {
        name, targetState, status, cves,
        scope: .scope.imageScope
      }]'

    Expect an approved false positive for CVE-2022-42889 and an approved deferral for CVE-2021-44228, both scoped to mfoster/shop-api with tag .*.

  4. Explain the RBAC split. A requester token cannot approve:

    echo "Requester role cannot approve. Expect not authorized:"
    curl -sk -o /tmp/vex-approve-denied.json -w "HTTP %{http_code}\n" \
      -H "Authorization: Bearer ${TEAM_A_TOKEN}" \
      -H "Content-Type: application/json" \
      -X POST "https://${CENTRAL_ROUTE}/v2/vulnerability-exceptions/does-not-exist/approve" \
      -d '{"comment":"demo"}'
    jq -r '.message // .' /tmp/vex-approve-denied.json
  5. In Vulnerability Management > Exception Management (or the deferred filter on Results), show that CVE-2021-44228 can be hidden as deferred while the exception is in force.

What they should notice:

  • OpenVEX in Git, exceptions in RHACS, approval as a separate audit step

  • Tag regex .* covers future shop-api tags

  • Prod-mirror is a different image name. Copying the digest does not copy the exception

Presenter tip: If Exception Management is empty, re-run bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/vex2acs.sh ~/rhacs-demo/basic-setup/vulnmgmt/vex/shop.openvex.json --approve after source of env.sh. HTTP 409 means they already exist.

Part 4: Stages, CI, and CVE impact

Know

Context: One application, many component images, three stages, and registry copies between them. The customer still needs to know which version is where, and which running software a given CVE hits.

R1 in this lab:

  • Stages are namespaces: demo-dev, demo-stage, demo-prod, labeled stage= and team=

  • Prod pulls quay.io/mfoster/prod-mirror-shop-api:1.0.0 (flat name, not prod-mirror/shop-api)

  • RHACS 5.0 tracks name plus digest. Same digest, two names, two records

  • Collections: shop-all, shop-prod, team-a-everything, platform-base-images

  • Watched images cover artifacts that are built but not deployed yet

  • There is no first-class application-version object. Labels such as app.version=1.0.0 belong to the customer’s model

R5 in this lab:

  • roxctl image check, image scan, image sbom, deployment check

  • REST for collections, exceptions, base images, watched images, RBAC

  • Policies as SecurityPolicy CRs

The umbrella question (D8): given a CVE ID, which deployments, which stages, which layer, is it fixable, and is it already deferred?

Show

What I say:

"RHACS is not your CMDB for application versions. It is the image-level truth. Your GitOps labels and collections answer 'what is in prod.' One CVE query then answers who is hit, in which stage, and who owns the layer."

What I do:

  1. Show the three stages and prod-mirror image name:

    oc get pods -n demo-dev -l app=shop
    oc get pods -n demo-stage -l app=shop
    oc get pods -n demo-prod -l app=shop
    oc get deploy shop-api -n demo-prod -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
  2. Same digest, two names:

    rox -G --data-urlencode 'query=Image:r/.*shop-api' --data-urlencode 'pagination.limit=50' \
      "https://${CENTRAL_ROUTE}/v1/images" | \
      jq '[.images[]? | {name: (.name.fullName // .name), id}]'

    "Prod uses prod-mirror-shop-api. That is a different image name with the same digest as shop-api:1.0.0."

  3. Collections and deployment labels:

    rox "https://${CENTRAL_ROUTE}/v1/collections" | \
      jq '[.collections[] | select(.name | test("shop-|team-a-|platform-base")) | {name, description}]'
    oc get deployments -n demo-dev -l app=shop --show-labels
  4. Watched images (bases from setup). Optionally watch the 1.1.0 release candidate:

    rox "https://${CENTRAL_ROUTE}/v1/watchedimages" | jq '.watchedImages'
    rox "https://${CENTRAL_ROUTE}/v1/watchedimages" -X POST \
      -H 'Content-Type: application/json' \
      -d '{"name":"'"${REG}"'/shop-api:1.1.0"}'
  5. Scan and SBOM for pipeline consumption (counts are live, not a script):

    roxctl image scan --endpoint "${ROX_ENDPOINT}" \
      --token-file <(echo "$ROX_API_TOKEN") \
      --image "${REG}/shop-api:1.0.0" --force --insecure-skip-tls-verify \
      -o json | jq '.result.summary'
    
    roxctl image sbom --endpoint "${ROX_ENDPOINT}" \
      --token-file <(echo "$ROX_API_TOKEN") \
      --image "${REG}/shop-api:1.0.0" --insecure-skip-tls-verify \
      | jq '{spdxVersion, packages_count: (.packages | length)}'
  6. Deployment pre-check as an Argo CD gate:

    roxctl deployment check --endpoint "${ROX_ENDPOINT}" \
      --token-file <(echo "$ROX_API_TOKEN") \
      -f ~/demo-applications/vulnmgmt/demo-dev/shop-api.yaml \
      --insecure-skip-tls-verify -o json 2>/dev/null | jq '{
        summary,
        policies: [.results[].violatedPolicies[]? | {policy: .name, severity, breaks_build: .failingCheck}][:8]
      }' || true

    If that yaml is not under ~/demo-applications/vulnmgmt, use the sibling clone next to rhacs-demo.

  7. Answer "which running software is affected by CVE-2021-44228?":

    bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/cve-impact.sh CVE-2021-44228
    bash ~/rhacs-demo/basic-setup/vulnmgmt/scripts/cve-impact.sh CVE-2021-44228 --team team-a
    rox "https://${CENTRAL_ROUTE}/v2/vulnerability-exceptions" | \
      jq '.exceptions[] | select(.cves[]? == "CVE-2021-44228") |
          {status, targetState, expiry: .deferralRequest.expiry}'

    "Expect shop-api in demo-dev, demo-stage, and demo-prod. layerIndex on the vulnerable log4j-core is the application COPY, not the platform base. The deferral from Part 3 is still APPROVED."

What they should notice:

  • Stages are namespaces and labels, not a built-in RHACS version object

  • prod-mirror-shop-api is a sibling repository name, not a nested path

  • One CVE query plus exceptions plus layer index is the customer’s daily workflow

  • Every step is scriptable. The human step is the assessment stored in OpenVEX

Part 5: Dev teams and branch environments

Know

Context: Developers should not live in the Central UI. They need their namespaces, a request path for exceptions, and vulnerabilities in the OpenShift console they already use.

R6 in this lab:

  • Access scope team-a-namespaces matches team=team-a

  • Role demo-team-a-dev uses the Vulnerability Management Requester permission set, scoped to that access scope

  • Role demo-vuln-approver uses the Approver permission set with unrestricted scope

  • The OpenShift console Security tab is the ACS plugin on the deployment

  • Reports can use the team-a-everything collection

Optional D9: Branch CI creates short-lived namespaces. Sensor sees them without onboarding. After delete, the deployment is gone. The image can remain searchable if it was watched or still within retention.

Show

What I say:

"Dev teams stay in OpenShift. They see their own namespaces. They can request a deferral. They cannot approve it. Branch environments appear when Argo CD creates them and disappear when the namespace is deleted."

What I do:

  1. Show the demo roles:

    rox "https://${CENTRAL_ROUTE}/v1/roles" | \
      jq '[.roles[] | select(.name | startswith("demo-")) |
           {name, permissionSetId, accessScopeId}]'
  2. Show scoped visibility with the team-a token (platform workloads should be absent):

    curl -sk -H "Authorization: Bearer ${TEAM_A_TOKEN}" \
      "https://${CENTRAL_ROUTE}/v1/deployments" | \
      jq '[.deployments[] | {name, namespace}]'
  3. Open the OpenShift console: {openshift_cluster_console_url}[window=blank]

    Navigate to Workloads > Deployments, project demo-dev, deployment shop-api, then the Security tab.

    "The developer never had to open Central to see these CVEs."

  4. Optional: Vulnerability Management > Vulnerability Reporting. Create a report on collection team-a-everything. The CSV is team-a only.

  5. Optional branch environment (skip if short on time):

    oc create namespace demo-branch-feature-x --dry-run=client -o yaml | oc apply -f -
    oc label namespace demo-branch-feature-x demo=vulnmgmt stage=branch team=team-a --overwrite
    cat <<EOF | oc apply -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: shop-api
      namespace: demo-branch-feature-x
      labels: {app: shop, component: api, app.version: "1.0.0-feature-x", team: team-a}
    spec:
      replicas: 1
      selector: {matchLabels: {app: shop, component: api}}
      template:
        metadata:
          labels: {app: shop, component: api, app.version: "1.0.0-feature-x", team: team-a}
        spec:
          containers:
          - name: api
            image: ${REG}/shop-api:feature-x-abc1234
            command: ["sleep", "infinity"]
            securityContext:
              allowPrivilegeEscalation: false
              runAsNonRoot: true
              capabilities: {drop: ["ALL"]}
              seccompProfile: {type: RuntimeDefault}
    EOF

    Wait until the deployment is listed in Results. team-a-everything includes it because of team=team-a. Then delete:

    oc delete namespace demo-branch-feature-x
    rox -G --data-urlencode 'query=Image Tag:r/feature-.*' --data-urlencode 'pagination.limit=50' \
      "https://${CENTRAL_ROUTE}/v1/images" | \
      jq '[.images[]? | (.name.fullName // .name)]'

What they should notice:

  • Scoped tokens hide demo-platform

  • Console Security tab is the daily view

  • Branch namespaces need no manual RHACS onboarding

Presenter tip: If TEAM_A_TOKEN is empty, script 10 did not persist it. Re-run bash ~/rhacs-demo/basic-setup/10-deploy-vulnmgmt-demo.sh and source ~/.bashrc.

Requirements coverage and known gaps

Req Capability Parts

R1

Namespaces, labels, collections, watched images, prod-mirror names

4, 5

R2

Exception API and OpenVEX bridge

3

R3

Base versus application layer, policy as code

1

R4

Red Hat VEX and content manifests

2

R5

roxctl, REST, SecurityPolicy CRs

1, 4

R6

Access scopes, console plugin, scoped reports

5

Gaps to say out loud:

  1. No native OpenVEX ingest (bridge script is the customer’s to own)

  2. Layer-type policy filters are Tech Preview and UI-only. Not in CRs yet

  3. No first-class application version entity. The customer owns that model

  4. Red Hat VEX filtering applies to Red Hat-shipped content. Third-party JARs use exceptions

  5. ROX_SCANNER_V4_RED_HAT_LAYERS_RED_HAT_VULNS_ONLY is not enabled by default