See the Story – Audit Log & Monitoring Correlation

Duration: ~30 minutes

Overview

Stand up a searchable audit store (Splunk), ship OpenShift API audit into it, then correlate an oc exec with runtime so you can tell a coherent story about who did what.

Why it matters

oc adm node-logs on a master (101-09) proves the file exists. Auditors and SOC analysts still ask: where is the durable, queryable copy? CIS audit-log-forwarding-enabled fails when those files never leave the control plane (101-10). A SIEM plus ClusterLogForwarder is that copy: identity, verb, object, and time in one place, joined to RHACS and container logs.

What does it solve

  • Audit stuck on masters with no hunt interface

  • Blind spots between “API says X” and “cluster measured Y”

  • Slow incident narratives that re-debate timelines

  • Missed privilege misuse that never lights a metric alert alone

Your Mission

Deploy Splunk as the lab SIEM, forward kube-apiserver audit to it, generate a suspicious exec, and hunt that event in Search—then join runtime. You already found exec on the node in 101-09; this lab puts the same event in a database you can query.

Prerequisites

  • oc logged in with permission to create namespaces, operators, and ClusterRoles (lab-user / cluster-admin)

  • jq and curl on the bastion

  • Storage for Splunk PVCs (about 20Gi + 5Gi)

Click each step only if you need a hint.

Part A: SIEM and audit forwarding

Splunk’s first boot often takes 10–25 minutes. Start install.sh, then keep reading Why it matters / Detection Logic while it rolls.
Deploy Splunk and forward OpenShift audit

Scripts live in this roadshow repo (vendored from rhacs-demo/splunk-setup, plus ClusterLogForwarder wiring). install.sh deploys Splunk, installs the RHACS add-on when ROX_API_TOKEN is set, then creates an HEC token and a ClusterLogForwarder that ships audit to Splunk.

cd ~/openshift-security-roadshow/setup/splunk-lab
SPLUNK_RUN_CLEAN_FIRST=false ./install.sh
End of the run prints Splunk Web URL, admin password, HEC endpoint, and example searches. First boot waits on deployment/splunk in namespace splunk.
Confirm Splunk Web and the collector

Open Splunk Web from the route. Collector pods appear in openshift-logging after the Cluster Logging Operator reconciles the forwarder.

oc get route splunk-web -n splunk -o jsonpath='https://{.spec.host}{"\n"}'
echo "admin password:"
oc -n splunk get secret splunk-auth -o jsonpath='{.data.password}' | base64 -d; echo
oc get clusterlogforwarder -A 2>/dev/null || true
oc get pods -n openshift-logging 2>/dev/null || true
Sign in as admin. Home may already be the RHACS Security Operations Dashboard. You will hunt kube-apiserver audit in Search & Reporting, not only that dashboard. If there is no ClusterLogForwarder yet, Part C still works via push-recent-audit.sh.

Part B: Generate a suspicious exec

Create project 201-06-a-correlation

Stay in this project for the sleeper pod.

oc new-project 201-06-a-correlation
Now using project "201-06-a-correlation".
Run pod sleeper

A long-running pod gives you a stable exec target.

oc run sleeper --image=registry.access.redhat.com/ubi9/ubi -- sleep 600
pod/sleeper created. Wait until it is Running before you exec.
Exec id and uname on sleeper

This is the suspicious action. Run it so audit records connect / exec.

oc exec sleeper -- sh -c 'id; uname -a'
You should see a uid and kernel string.
Exec on sleeper a second time

Repeated exec looks like probing, not a one-off debug session.

oc exec sleeper -- sh -c 'id; uname -a'
Same output. Two audit events should exist.

Part C: Hunt in Splunk

Seed Splunk with recent kube-apiserver audit

ClusterLogForwarder collectors can take several minutes. This copies the last lines of each master’s kube-apiserver/audit.log into HEC so Search is not empty for the demo. It uses oc adm node-logs (same path as 101-09), not a file on the bastion.

cd ~/openshift-security-roadshow/setup/splunk-lab
./push-recent-audit.sh
Posted from <master>. Re-run after more execs if the first window was too early.
Search Splunk for sleeper exec

Splunk Web → Search & Reporting. Set the time picker to Last 1 hour. Run:

index=main (sleeper AND (exec OR connect))

Also try the sourcetype the HEC token sets:

index=main sourcetype="kube:apiserver:audit" sleeper
Expand an event and read user.username (or user.groups), objectRef.namespace, objectRef.name (sleeper), objectRef.subresource (exec), verb (create / connect), and sourceIPs. That is the API half of the story.
Correlate with RHACS in the same SIEM (optional)

If install.sh created the RHACS Splunk notifier (ROX_API_TOKEN was set), Central also writes alerts into HEC.

index=* sourcetype="stackrox-*" sleeper
API audit names the Kubernetes user and verb. RHACS names the process inside the container. Same pod, two sensors. Empty stackrox-* means the notifier or add-on inputs have not run yet—kube-apiserver hits are still enough for this lab.

Part D: Join runtime

Read sleeper logs from the last two minutes

Join the audit timestamp to what the container printed.

oc logs sleeper -n 201-06-a-correlation --since=2m
The sleep image may log little; empty logs still prove you checked the runtime side. In RHACS, look for a matching process-exec finding.
Confirm the node file still matches Splunk

Spot-check that the SIEM did not invent the event. Use oc adm node-logs, not grep on the bastion—the audit file is on the control plane.

MASTER=$(oc get nodes -l node-role.kubernetes.io/control-plane -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -z "$MASTER" ]; then
  MASTER=$(oc get nodes -l node-role.kubernetes.io/master -o jsonpath='{.items[0].metadata.name}')
fi
oc adm node-logs "$MASTER" --path=kube-apiserver/audit.log | grep sleeper | grep exec | tail -3
JSON lines with sleeper and exec. Same user and pod you saw in Splunk.

Detection Logic Draft

Rule: IF (exec to prod namespace pod) AND (user not in admin group) THEN alert severity=high.

Data points:

  • Audit (Splunk): user, verb=create/connect, subresource=exec, objectRef.name, sourceIPs

  • RBAC: subject group membership

  • Namespace label: env=prod

  • SIEM enrichment: cluster and namespace fields on forwarded events so queries stay simple

  • RHACS: process-exec / violation on the same pod name and time window

Debrief

Audit shows API intent; logs, metrics, and RHACS show runtime. Correlate on namespace, pod, user, verb, and time to spot pivots that neither stream alone catches.

What “configured for OpenShift audit logging” means here:

  • Store — Splunk (HEC + index main) instead of grepping a node file during an incident

  • Ship — ClusterLogForwarder inputRefs: [audit] to that HEC (CIS OpenShift 1.2.21)

  • Enabled — API audit profile is not None (CIS 3.2.1); you do not turn that off

  • Seed — push-recent-audit.sh so the lab hunt does not wait on collector rollout

  • Evidence — leave Splunk and the forwarder up; 201-07 re-scans ocp4-cis so 1.2.21 / 3.2.1 show PASS

What breaks without this:

  • Siloed audit vs runtime → missed exec campaigns and privilege grants

  • No join keys → alerts without investigation context

  • CIS audit-log-forwarding-enabled (1.2.21) stays FAIL because the files never leave the masters

  • No 201-07 rescan → Splunk works, but the CIS report still shows the 101-10 FAIL

ship audit immutably, align clocks, query by identity and verb, and build anomaly rules (for example non-admin exec in prod).
this lab is the foundation for later ACS/IR correlation—not a replacement for runtime sensors. Splunk first boot is slow. Do not tear down Splunk or the ClusterLogForwarder yet — 201-07 re-scans ocp4-cis against this forwarder. Module cleanup only deletes the sleeper project.
giphy

Cleanup

This module’s cleanup removes project 201-06-a-correlation only. Splunk and the audit ClusterLogForwarder stay for 201-07. After 201-07, lab cleanup removes those too (setup/splunk-lab/clean.sh).

cd ~/ocp5-rhacs-showroom
bash setup/lab-cleanup.sh --module 201-06