CI/CD and Automation

1. Module Goals

2. Implement Policy-as-Code in ACM with OpenShift GitOps

Here, you will manage ACS security policies via RHACM and OpenShift GitOps. First, you must create an ArgoCD controller in RHACM.

2.1. RHACM Console Access

The RHACM console is available in the OpenShift cluster console at: {openshift_console_url}

Administrator login is available with:

Username:

{openshift_admin_user}

Password:

{openshift_admin_password}

Navigate to the Cluster dropdown menu and then select All Clusters:

05 acm 00

2.2. Implement Security Policy-as-Code

Procedure

  1. Navigate to Applications from the left side menu:

05 acm 02

  1. Click Create application, select ArgoCD ApplicationSet-Push Model:

  2. Configure the Application:

    • Name: Give the application the name pac-custom-policies:

pac-custom-policies

  1. Under the Argo server, select Add Argo Server:

05 acm 03

  1. Enter the following information:

    • Name: openshift-gitops

    • Namespace: openshift-gitops

    • ClusterSet: default:

  2. Click "Add," then make sure to select the created Argo server:

05 acm 04

PaC Custom Policies on GitHub

  1. Click "Next"

  2. In the Template tab, select Git Repository and enter the URL of the GitHub repository containing the custom policies - URL: https://github.com/mfosterrox/skupper-security-demo.git:

  3. Select "Main":

  4. Select "PaC-custom-policies":

  5. Enter the remote namespace of "stackrox":

05 acm 05

  1. Click "Next TWICE":

  2. Set the following in the Placement tab:

    • Cluster sets: default:

  3. Under Label expressions, click add label and select the following:

    • Label: name

    • Operator: equals any of

    • Values: local-cluster:

05 acm 06

  1. Click "Next":

  2. Click "Submit":

  3. Click on "Topology" and wait for the policies to go green:

05 acm 06.5

3. Ensure That the Policies Are Properly Configured

Let’s run the check again to check if the policies were successful.

Procedure

  1. Perform an image security scan using roxctl to check for policy violations:

roxctl -e $ROX_CENTRAL_ADDRESS:443 image check --image $QUAY_URL/$QUAY_USER/frontend:0.1

Expected: Two failing APK enforce policies plus GitOps-managed policies from the previous section.

You should see a second APK policy violation from the externally managed GitOps policy.

3.1. Check on Your Policies

Procedure

  1. Go to the RHACS dashboard and head to the Platform Configuration → Policy Management dashboard:

  2. Search by Policy the alpine:

05 acm 07

Notice how the policies are System, Locally managed, and Externally managed?

What’s the difference?

  • System policies come with RHACS by default

  • Locally managed policies are created in RHACS or via the API

  • Externally managed policies are managed via GitOps and policy as code

  1. Delete the locally managed "conflicting" policies by running the following command:

POLICY_ID=$(curl -X GET \
  -H "Authorization: Bearer $ROX_API_TOKEN" \
  -H "Content-Type: application/json" \
  https://$ROX_CENTRAL_ADDRESS/v1/policies | jq -r '.policies[] | select(.name=="Alpine Linux Package Manager in Image - Enforce Build") | .id')

curl -X DELETE \
  -H "Authorization: Bearer $ROX_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "id": "$POLICY_ID"
      }' \
  https://$ROX_CENTRAL_ADDRESS/v1/policies/$POLICY_ID

Nice job!

4. Implementing Security Checks into Your CI/CD Pipeline

RHACS integrates into CI/CD via roxctl and the admission controller for image scanning, secret handling, and compliance checks. Best practices include RBAC on pipelines, minimal base images, dependency pinning, and policy-as-code.

5. Review a Basic CI/CD Pipeline

Review a simple OpenShift Pipelines (Tekton) workflow that scans images with roxctl before deploy.

Procedure

  1. Log in to the OpenShift console at {web_console_url} (same credentials as earlier modules). Select local-cluster, then open Pipelines.

  1. Apply the pipeline resources:

oc apply -f $TUTORIAL_HOME/openshift-pipelines/ --recursive

Expected: rox-pipeline, roxsecrets, and Tekton tasks are created.

Find rox-pipeline in project pipeline-demo, or switch the project filter to pipeline-demo.

05 cd 03

Start a pipeline run:

  1. Click the three dots on the right side of the screen and select start:

05 cd 05 
This pipeline works by taking an image that you have created and passing it through a "roxctl image scan" and a "roxctl image check" process.

  1. Let’s use the frontend:0.1 image that we’ve been using all day. Run the following and input it into the image bar:

echo $QUAY_URL/$QUAY_USER/frontend:0.1
05 cd 06
The pipeline should fail.

Why do you think the pipeline failed?

Let’s look at the logs.

  1. Take a look at the log snippet on the bottom right of the page:

05 cd 09

Expected: Initial failure—Could not resolve host: ROX_CENTRAL_ENDPOINT and missing roxctl indicate incorrect pipeline secret variables.

Inspect the pipeline secret template:

  1. Run the following in the terminal:

cat $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

The secret requires rox_central_endpoint and rox_api_token. Update placeholders with your Central route and API token:

  1. Run the following in the terminal to update the RHACS Central address and API Token:

ACS_URL=$(oc -n stackrox get route central -o jsonpath='{.spec.host}')
ACS_URL_PORT=$(echo "$ACS_URL" | sed 's/$/:443/')
sed -i "s|ROXCTL_CENTRAL_ENDPOINT|$ACS_URL_PORT|g" $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml
sed -i "s|API_TOKEN|$ROX_API_TOKEN|g" $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

  1. Apply the updated secret:

oc apply -f $TUTORIAL_HOME/openshift-pipelines/secrets/rox-secrets.yml

5.1. Let’s Try Again

Procedure

  1. Go back to the pipelines view:

05 cd 12

  1. Click the Actions dropdown and select Rerun. Then relax for a few seconds and grab a sip of water:

giphy
05 cd 13

What happened? Did you expect this behavior?

  1. Check the log snippet in the bottom right of the page.

Expected: Pipeline fails on enforced policy—ERROR: failed policies found: 1 policies violated (e.g. Ubuntu Package Manager in Image - Build-time).

So our policy-as-code example blocked the pipeline from executing successfully.

6. What Would You Do?

  • How would you implement roxctl into your pipelines?

  • Would you have enforcement off at the beginning?

  • Are there any organizational policies that should be enforced in all pipelines?

7. Security Exercise: ACME Incident Response

At 02:14 AM Morgan paged Sarah to trace how the attacker got shell access. The answer is in the audit logs—and it implicates a CI/CD service account.

You are participating in the ACME Financial Services security incident workshop. Sarah, a senior OpenShift Platform Engineer, and Morgan from Infosec are responding to a breach in the payment-gateway production namespace. An attacker used a compromised CI/CD service account to exec into the payment-gateway pod, deploy a hidden toolkit (/tmp/.hidden_toolkit/exfil.sh), and beacon outbound to a command-and-control server.

Each ACS module ends with a hands-on exercise that advances this incident response story using RHACS and OpenShift platform tools.

7.1. Scenario

I’m querying control-plane audit logs for exec events in our namespace. Morgan needs to know if this is bigger than one pod.
— Sarah
Platform Engineering

7.2. Your task

The steps are there for guidance. If you need help, expand the step and compare your approach to the lab commands.

Step 1 — Switch to your payment gateway namespace
Use your assigned workshop namespace suffix 1.

  • Payment gateway (breached workload): payment-gateway-1

  • Vulnerable Socket.IO demo: vulnerable-workload-1

Switch to your payment gateway namespace before running commands:

oc project payment-gateway-1

If your administrator has not yet provisioned the environment, ask them to run ./setup/deploy-incident-env.sh from the roadshow repository (see setup/README.adoc).

Step 2 — Search audit logs for the compromised exec

Search audit logs on control plane nodes for the compromised exec:

namespace=$(oc project --short)

for master in $(oc get nodes --selector node-role.kubernetes.io/master --output name); do
  echo "=== Searching logs on ${master} ==="
  oc adm node-logs "${master}" --path=kube-apiserver/audit.log \
    | jq --arg ns "${namespace}" 'first(select(.objectRef.namespace == $ns
          and .objectRef.name == "payment-gateway"
          and .verb == "create"
          and .objectRef.subresource == "exec"))'
done
Step 3 — Interpret the audit log output

Review the output. Key fields to identify:

  • user.username — should be system:serviceaccount:payment-gateway-1:cicd-pipeline-runner-sa

  • requestURI — URL-encoded payload with SHADOW_PIVOT, /tmp/.hidden_toolkit/exfil.sh

  • responseStatus.code — 101 (WebSocket upgrade = successful exec)

  • annotations — RBAC binding that allowed the exec

Incident investigation

7.3. What you learned

  • Audit logs attribute exec actions to the exact service account and source IP.

  • Over-permissioned CI/CD credentials are a common real-world attack path.

  • Pipeline automation accounts should not have live pod exec permissions in production.

8. Summary

THUMBS UP!

In this lab, you reviewed a basic CI/CD use case that implemented the build policy from the previous module. You then adjusted the policy so the pipeline would succeed.

Congrats! On to Compliance!

giphy

9. Cleanup

Before moving to the next module, run the lab cleanup script to reset transient resources from this module.

cd ~/openshift-security-roadshow
bash setup/lab-cleanup.sh --module 05