Logs Don’t Lie – Tracing Who Did What (Audit Forensics)

Duration: ~20 minutes

Overview

Know where OpenShift writes API audit logs, which CIS controls that satisfies, then generate an oc exec and pull the event from the Kubernetes API server audit file.

Why it matters

When something goes wrong, audit logs show the API calls: user, verb, object, time, source. CIS Kubernetes expects that trail to exist, rotate, and cover sensitive verbs such as exec. That is how you reconstruct an incident instead of arguing from memory.

What does it solve

  • Reduces downtime in incident reconstruction

  • Prevents disputes over actions

  • Supports least privilege validation

  • Gives auditors a concrete answer for “where are the API audit logs?” and “is exec recorded?”

Your Mission

Someone may already have exec’d into a pod. Your mission: know the log locations CIS cares about, make an exec yourself, then pull the audit trail fast enough to name the actor, pod, and time. Complete Part A, then Part B.

Deeper correlation in Splunk (SIEM + ClusterLogForwarder) with RHACS is 201-06. Compliance Operator scans are 101-10.

Click each step only if you need a hint.

Part A: CIS controls and where the logs live

OpenShift writes three API audit streams on every control plane node. oc adm node-logs reads them without SSH. You need permission to get nodes/log (cluster-admin or the roadshow node-log-viewer ClusterRole).

List kube-apiserver audit files on the masters

On-disk paths are on the control plane node, not the bastion. The oc adm path argument is relative to /var/log/.

What On the node How you read it from the bastion

Kubernetes API server (exec, pods, Secrets)

/var/log/kube-apiserver/audit.log (plus rotated audit-*.log)

oc adm node-logs <node> --path=kube-apiserver/audit.log

OpenShift API server (Routes, projects, operators)

/var/log/openshift-apiserver/

oc adm node-logs <node> --path=openshift-apiserver/

OAuth API server (tokens, logins)

/var/log/oauth-apiserver/

oc adm node-logs <node> --path=oauth-apiserver/

Node journal

journald on the node

oc adm node-logs <node> (no --path)

Cluster Logging / Splunk

indexed store (HEC, sourcetype=kube:apiserver:audit)

201-06

List the kube-apiserver audit files that actually exist:

oc adm node-logs --role=master --path=kube-apiserver/ | head -40
You should see audit.log and rotated audit-YYYY-…log files. That is the CIS “audit log path is set” evidence.
Read kube-apiserver audit-log-path and CIS check names

Kubernetes CIS (kube-apiserver) is what this module maps to. OpenShift CIS (ocp4-cis in Compliance Operator) asks the same questions with OpenShift rule names.

CIS Kubernetes What an auditor asks What “good” looks like here

1.2.22

Is --audit-log-path set?

Files under /var/log/kube-apiserver/ (previous step)

1.2.23

Are logs retained long enough (--audit-log-maxage)?

Retention days in the API server config; not only “we have a file today”

1.2.24

Are enough rotated files kept (--audit-log-maxbackup)?

Several audit-*.log files in the listing

1.2.25

Do files rotate at a max size (--audit-log-maxsize)?

OpenShift typically rotates around 100–200 MiB

3.2.1

Is there a minimal audit policy?

OpenShift ships a policy; API audit events exist

3.2.2

Does the policy cover key security events?

exec, portforward, Secret access, and RBAC changes are logged

Local files rotate and age out. CIS also expects you to forward them (SIEM / Cluster Logging). Disk on the master is not the archive.

oc get kubeapiserver cluster -o jsonpath='{.spec.observedConfig.apiServerArguments.audit-log-path}{"\n"}' 2>/dev/null || true
oc get compliancecheckresults.compliance.openshift.io -n openshift-compliance 2>/dev/null | grep -iE 'audit|1.2.22|3.2' | head || echo 'Compliance Operator results not in this cluster -- use the table above.'
Treat any FAIL rows as a teaser. 101-10 filters the audit-log rules and prints what to change.

Part B: Generate exec and hunt it

Create a pod, exec into it, then search the Kubernetes API audit log. Metadata-level audit still records who and which pod; the command string is present only when the policy logs the request body.

Create project 101-09-a-audit and deploy audit-sleeper

You need a running pod so the next step can oc exec into it.

oc new-project 101-09-a-audit || oc project 101-09-a-audit
oc create deployment audit-sleeper --image=registry.access.redhat.com/ubi9/ubi -- sleep infinity
oc rollout status deployment/audit-sleeper --timeout=90s
Rollout should complete. Pod Running.
Exec id and uname in audit-sleeper

These two execs are the events you will hunt in the audit log.

oc exec deploy/audit-sleeper -- id
oc exec deploy/audit-sleeper -- uname -n
You should see a UID and a hostname. Give the log a few seconds to catch up before the next step.
Grep kube-apiserver audit.log for this namespace and exec

This reads /var/log/kube-apiserver/audit.log on the first control plane node. Give the exec a few seconds to land in the file. This is one pipeline; count greater than 0 means your exec is in the file.

MASTER=$(oc get nodes -l node-role.kubernetes.io/master -o jsonpath='{.items[0].metadata.name}')
echo "Reading audit.log on ${MASTER}"
sleep 5
oc adm node-logs "${MASTER}" --path=kube-apiserver/audit.log \
  | grep -F '101-09-a-audit' \
  | grep -E '"exec"|portforward' \
  | tee /tmp/lab-101-09-audit.json \
  | wc -l
Count greater than 0. If 0, wait and re-run, or try another master (oc get nodes -l node-role.kubernetes.io/master).
jq exec events from /tmp/lab-101-09-audit.json

Look for your user, namespace 101-09-a-audit, and pod name audit-sleeper-…​.

jq -c 'select(.objectRef.subresource=="exec") | {time:.requestReceivedTimestamp,user:.user.username,ns:.objectRef.namespace,pod:.objectRef.name,verb:.verb,sourceIPs:.sourceIPs}' /tmp/lab-101-09-audit.json

Fallback if jq complains about concatenated objects:

grep -F '"exec"' /tmp/lab-101-09-audit.json | head

Look for your user, namespace 101-09-a-audit, and pod name audit-sleeper-….

jq portforward events from the same file

You did not open a port-forward. Empty output is a pass—this query is how you would catch a tunnel.

jq -c 'select(.objectRef.subresource=="portforward") | {time:.requestReceivedTimestamp,user:.user.username,ns:.objectRef.namespace,pod:.objectRef.name}' /tmp/lab-101-09-audit.json || true
Count unique exec usernames in this slice

This is who appears as the actor on exec in the saved JSON.

jq -r 'select(.objectRef.subresource=="exec") | .user.username' /tmp/lab-101-09-audit.json | sort | uniq -c
Your current user should appear at least once.
Write the one-line incident summary

Fill this from the jq output: who (user), what (exec on which pod), when (time), from where (sourceIPs). That is the CIS-relevant evidence: sensitive API actions are recorded and attributable.

At <time>, <user> from <sourceIP> exec'd into pod <pod> in namespace 101-09-a-audit.

Debrief

You mapped Kubernetes CIS audit controls to OpenShift file paths, listed real kube-apiserver audit files, generated an exec, and extracted who/when/which pod from /var/log/kube-apiserver/audit.log.

What breaks without this:

  • No retention or shipping → you cannot answer who/what/when after an incident

  • Ignoring exec/port-forward → classic post-compromise footholds stay invisible

  • Knowing only the node file and never a SIEM → you fail the auditor’s “where does it live?” question

keep OpenShift audit on, know the three /var/log/…-apiserver/ trees, aggregate immutably off the node (CIS intent of 1.2.23–1.2.25 plus log forwarding), correlate with RHACS/runtime in later modules, and alert on anomalous verbs for sensitive namespaces.
audit shows API intent; it complements (does not replace) runtime detection. Metadata logs prove the exec; RequestResponse may also capture the command. Confirm retention and SIEM forwarding before you need them. Next: 101-10 grades those CIS rows (path, rotation, forwarding) as PASS/FAIL.
giphy

Cleanup

Before moving to the next module, run the lab cleanup script to reset transient resources from this module.

cd ~/ocp5-rhacs-showroom
bash setup/lab-cleanup.sh --module 101-09