Skip to main content

Pattern: Kubernetes Pod RCA Agent

This page is also the pattern template. Every use case in the library follows this same shape. Copy it, fill it in, ship it. Your Draft agent can populate this structure automatically from a build's notes.

Pillar3 · Agentic Engineering
DomainContainers / Kubernetes / SRE
AutonomyRead + diagnose autonomously · recommend, don't act (human approves fixes)
Repoopenagentix/k8s-pod-rca-agent

The problem​

A pod is crash-looping at 2 a.m. An engineer wakes up, runs the same handful of commands they always run — kubectl describe, kubectl logs, check events, check the node, check recent deploys — pieces the story together, and writes it up. It's repetitive, it's slow, and the quality depends on who's on call and how awake they are.

This is a perfect first agent: the diagnostic steps are well-known, bounded, and mostly read-only. The judgment (what to do) stays with the human.

What the agent does​

  1. Detects (or is pointed at) a problem pod.
  2. Gathers the evidence a good SRE would: pod status & events, container logs, restart counts, resource limits vs. usage, recent rollouts, node conditions.
  3. Correlates the signals into a root-cause hypothesis (e.g. OOMKilled → memory limit too low; ImagePullBackOff → bad tag/secret; CrashLoopBackOff → failing readiness probe).
  4. Produces a concise RCA report with the evidence and 1–3 recommended actions, ranked.
  5. Posts the report to Slack for a human to approve — it does not apply fixes on its own.

Trigger​

  • Interactive: an engineer asks in Slack ("RCA this pod: payments-7d9f...").
  • Event-driven: fired by an alert (e.g. Alertmanager KubePodCrashLooping).

Tools & integrations​

  • kubectl (scoped, read-only RBAC) against the target cluster
  • Metrics source (Prometheus / Datadog) for resource usage
  • Slack (for the report + approval)
  • Optional: the deploy history source (ArgoCD / Git) to spot "what changed"

Architecture​

Guardrails & human oversight​

This is the part that makes it TrinityOps and not a demo:

  • Read-only by default. The agent's cluster credentials can look, not touch. Fixes are a separate, human-approved step.
  • Scoped access. RBAC limited to the relevant namespaces; no cluster-admin.
  • Evidence, not assertions. Every conclusion in the report cites the signal it came from, so a human can verify quickly.
  • Approval gate. No action reaches production without a person clicking approve.
  • Audit trail. Every run logs what it looked at and what it recommended.

How to build it (starter path)​

  1. Start read-only and interactive — just "summarize what's wrong with this pod." No actions.
  2. Add correlation rules for the top failure modes (OOMKilled, ImagePullBackOff, CrashLoopBackOff, failing probes, pending/unschedulable).
  3. Add the "what changed recently?" signal — it resolves a huge share of incidents.
  4. Add the Slack report + approval gate.
  5. Only then, and only if you want it, add optional one-click remediation behind explicit human approval.
  • Alert Auto-Resolver (Pillar 3) — the "act on it" companion, runbook-driven
  • Oncall Shift Handover (Pillar 3) — summarizes a shift's worth of these
  • Monitoring Correlation (Pillar 3) — broader cross-tool correlation

Built in public in the Agentix Garage. 👉 Get the next build in your inbox