See the Story – Audit Log & Monitoring Correlation
Duration: ~20 minutes
Overview
Correlate audit logs & monitoring signals.
Correlate API audit events with monitoring signals so you can tell a coherent story about who did what and what the cluster observed.
Why it matters
Audit logs alone and dashboards alone leave incident stories half-told. Correlating API verbs with monitoring signals lets you prove who did what while the cluster’s health signals confirm impact.
What does it solve
-
Blind spots between “API says X” and “cluster measured Y”
-
Slow incident narratives that re-debate timelines
-
Missed privilege misuse that never lights a metric alert alone
Your Mission
Connect audit evidence to monitoring signals so a suspected foothold’s actions and blast radius show up as one coherent story.
Prerequisites
-
Cluster logging or LokiStack configured (audit indexed)
-
Permissions to read audit indices
Click each step only if you need a hint.
Namespace & Pod
oc new-project 201-07-a-correlation
oc run sleeper --image=registry.access.redhat.com/ubi9/ubi -- sleep 600Suspicious Exec
Run the exec more than once so audit shows repeated probing, not a one-off mistake.
oc exec sleeper -- sh -c 'id; uname -a'
oc exec sleeper -- sh -c 'id; uname -a'Query Audit (Conceptual Command)
If using Loki:
{log_type="kube-apiserver"} |= `connect` |= `exec` |= `sleeper` | json| Look for these fields in the results. |
-
user.username -
objectRef.name/ namespace -
verb (
connect) +subresource=exec -
sourceIPs[]
Extract Fields (API Example)
# If you have direct file access (control plane), pseudo:
grep sleeper /var/log/kube-apiserver/audit* | grep exec | tail -1Parse JSON: identify timestamp, user, pod.
Metrics / Logs Correlation
Check pod logs around timestamp:
oc logs sleeper -n 201-07-a-correlation --since=2mIf using ACS runtime, find corresponding process execution alert conceptually.
Detection Logic Draft
Rule: IF (exec to prod namespace pod) AND (user not in admin group) THEN alert severity=high. Data Points: * Audit: user, verb=connect, subresource=exec * RBAC: subject group membership * Namespace label: env=prod * SIEM enrichment: cluster and namespace labels on forwarded audit events so queries stay simple
Debrief
Audit shows API intent; logs and metrics show runtime. Correlate on namespace, pod, user, verb, and time to spot pivots that neither stream alone catches.
What breaks without this:
-
Siloed audit vs runtime → missed exec campaigns and privilege grants
-
No join keys → alerts without investigation context
Controls that matter: ship audit immutably, align clocks, query by identity and verb, and build anomaly rules (for example non-admin exec in prod).
Quick facts: this lab is the foundation for later ACS/IR correlation—not a replacement for runtime sensors.
