Incident Response with Splunk Audit and RHACS

Table of Contents

Overview
Why it matters
What does it solve
Your Mission
Debrief
Cleanup

Duration: ~25 minutes

Overview

201-11 Repudiation: an exec or policy kill with no durable trail is an incident you cannot explain. 201-06 stood up Splunk and kube-apiserver audit. 301-04 made ACS kill a violating process.

This lab is the use of those two sensors together: generate (or reuse) a bad action, hunt it in Splunk (API audit + optional stackrox-*), join it to the ACS violation, and write a five-line timeline a SOC lead could hand to an incident commander.

Why it matters

ACS names the process. Audit names the Kubernetes user and verb. Neither stream alone answers “who exec’d dnf into which pod, and did runtime enforce fire?” CIS still cares that audit left the masters (1.2.21); IR cares that you can query it under time pressure.

What does it solve

Violations in ACS with no API identity
Splunk exec hits with no process name
Rewriting the timeline from memory after the pod is gone
Forensics theater (snapshots, chain-of-custody binders) before you can search

Your Mission

Confirm Splunk is up (reinstall if 201-07 cleanup tore it down), generate an exec against the 301-04 probe or a sleeper, hunt both indexes, and fill in the timeline table.

Prerequisites

Splunk + ClusterLogForwarder from 201-06, or re-run setup/splunk-lab/install.sh
RHACS Central (optional notifier into Splunk if ROX_API_TOKEN was set at install)
Something to hunt: leftover 301-04-runtime probe, or a sleeper in this lab

Click each step only if you need a hint.

Part A: SIEM still there?

Splunk route and password

oc get route splunk-web -n splunk -o jsonpath='https://{.spec.host}{"\n"}' 2>/dev/null || echo "Splunk missing — reinstall"

If the route is missing (typical after 201-07 cleanup):

cd ~/openshift-security-roadshow/setup/splunk-lab
SPLUNK_RUN_CLEAN_FIRST=false ./install.sh

Then:

oc get route splunk-web -n splunk -o jsonpath='https://{.spec.host}{"\n"}'
echo "admin password:"
oc -n splunk get secret splunk-auth -o jsonpath='{.data.password}' | base64 -d; echo

Part B: Generate the bad action

Exec into the 301-04 probe (or a sleeper)

Prefer the probe if 301-04 is still deployed—then ACS and audit line up on the same pod.

if oc get deploy pkgmgr-probe -n 301-04-runtime >/dev/null 2>&1; then
  POD=$(oc get pod -l app=pkgmgr-probe -n 301-04-runtime -o jsonpath='{.items[0].metadata.name}')
  echo "Using probe $POD"
  oc exec "$POD" -n 301-04-runtime -- /bin/bash -c 'dnf --version || true; echo IR-LAB-EXEC'
else
  oc new-project 301-05-ir || oc project 301-05-ir
  oc run sleeper --image=registry.access.redhat.com/ubi9/ubi -- sleep infinity
  oc wait pod/sleeper --for=condition=Ready --timeout=90s
  oc exec sleeper -- echo IR-LAB-EXEC
fi
date -u +%Y-%m-%dT%H:%M:%SZ

Note the UTC timestamp. If ACS runtime enforce is still on, the probe pod may be killed (that is evidence, not failure).

Seed Splunk if the forwarder is slow

cd ~/openshift-security-roadshow/setup/splunk-lab
chmod +x push-recent-audit.sh
./push-recent-audit.sh

Same helper as 201-06. Gives you kube-apiserver lines without waiting on the collector.

Part C: Hunt and join

API audit in Splunk

Search:

index=main sourcetype="kube:apiserver:audit" (pkgmgr-probe OR sleeper OR 301-04-runtime OR 301-05-ir) exec

Expand an event: user.username, objectRef.namespace, objectRef.name, objectRef.subresource (exec), verb (create / connect), sourceIPs. That is who called the API.

RHACS in Splunk (if the notifier is installed)

index=* sourcetype="stackrox-*" (pkgmgr-probe OR 301-04-runtime OR "Package Manager")

Empty stackrox-* means the notifier was not configured—open Violations in Central for the same window. You still have the audit half.

ACS console + pod logs

oc get route central -n stackrox -o jsonpath='https://{.spec.host}{"\n"}'
oc get events -n 301-04-runtime --sort-by='.lastTimestamp' 2>/dev/null | tail -20
oc get events -n 301-05-ir --sort-by='.lastTimestamp' 2>/dev/null | tail -20

Central Violations: policy, deployment, process, enforcement. Events: Killing / Started. Together with Splunk you can fill the table below.

Part D: Five-line timeline

Fill this in from the hunts (example answers in the hint).

# Question Your evidence (source)

#	Question	Your evidence (source)
1	When (UTC)?	Splunk `_time` / `date` from the exec step
2	Which Kubernetes user?	Audit `user.username` (not the container UID)
3	Which namespace / pod?	`objectRef` + ACS deployment
4	What process ran?	ACS violation (e.g. `dnf`) / `echo IR-LAB-EXEC`
5	Did runtime enforce fire?	ACS enforcement + OpenShift `Killing` event

When (UTC)?

Splunk _time / date from the exec step

Which Kubernetes user?

Audit user.username (not the container UID)

Which namespace / pod?

objectRef + ACS deployment

What process ran?

ACS violation (e.g. dnf) / echo IR-LAB-EXEC

Did runtime enforce fire?

ACS enforcement + OpenShift Killing event

Example filled row (collapsible)

14:02:11Z — lab-user — 301-04-runtime / pkgmgr-probe-… — dnf --version — yes, ACS runtime enforce killed the pod; Deployment recreated a replica.

Detection rule worth keeping: IF (exec in a prod-like namespace) AND (ACS package-manager violation within 60s) THEN page the on-call.

Debrief

IR on OpenShift is Splunk audit + RHACS, not a separate forensics appliance for the first hour. Preserve the SIEM and Central; snapshot the node later if you need disk.

What breaks without this:

ACS-only → you know dnf ran, not which API user
Audit-only → you know exec happened, not which binary
Tearing Splunk down after 201-07 and never reinstalling → 301 has no hunt UI

Cleanup only deletes 301-05-ir (not Splunk, not ACS policies).

Cleanup

Deletes project 301-05-ir if you created the sleeper. Does not uninstall Splunk or ACS.

cd ~/ocp5-rhacs-showroom
bash setup/lab-cleanup.sh --module 301-05