48.9k
An AI SRE is an LLM-based agent that investigates production alerts the way a human site reliability engineer would, querying logs, metrics, and traces, forming hypotheses, and proposing mitigations. At Wix, AI agents enrich 30,000+ alerts a month with 90% root-cause accuracy, while humans still make every remediation decision.
TL;DR
- An AI SRE agent automates the investigation step of incident response (evidence gathering, correlation, ranked hypotheses) and leaves the judgement step to humans.
- In ClickHouse's 2025 root-cause-analysis experiment, no frontier model reliably found injected root causes without human guidance; human-in-the-loop remains current best practice.
- AI SRE agents differ from classic AIOps: agentic investigation with follow-up queries versus statistical anomaly detection and event correlation.
- Agents run 6-27 queries per investigation, so their speed is bounded by the query latency, retention, and cardinality of the underlying telemetry store.
- Agents take over the repetitive first hour of an investigation; remediation decisions, capacity planning, and SLO work stay with humans.
What is an AI SRE?
An AI SRE is a large-language-model agent that performs the investigation tasks of site reliability engineering: it receives an alert, pulls evidence from logs, metrics, traces, and recent changes, forms ranked hypotheses about the cause, and proposes mitigations, executing them only with approval.
The term covers a family of products and internal tools that emerged through 2024-2025, also marketed as "AI on-call copilots" or "incident response agents." What distinguishes them from a chatbot bolted onto a dashboard is the agentic loop: the model calls tools, reads the results, and decides what to query next. The work they target is what the Google SRE book calls toil: manual, repetitive, automatable operational work that Google caps at 50% of an SRE's time. Alert triage and first-pass investigation are squarely inside that definition.
How do AI SRE agents investigate incidents?
An AI SRE agent investigates by running a tool-use loop in which every iteration is a real query against live telemetry.
- Trigger. An alert or page fires; the agent receives it with service, severity, and timing metadata.
- Gather evidence. The agent queries error rates, latency percentiles, log patterns, trace exemplars, and recent deployments around the alert window. ClickHouse's reference architecture for AI SRE measured 6-27 SQL queries per investigation.
- Hypothesize and verify. The model drafts candidate causes (a deploy, a saturated dependency, a bad config) and runs targeted queries to test each one.
- Report. It posts ranked hypotheses with evidence to the on-call channel. At Wix, Wild Moose agents deliver findings to Slack within about a minute of an alert firing, running parallel queries against a cluster ingesting 300 million events per minute.
{
"alert_id": "alrt-9017",
"investigated_at": "2026-07-28T04:12:44Z",
"queries_run": 14,
"top_hypothesis": "checkout p99 regression follows payments v3.2.1 deploy at 04:05",
"confidence": 0.82,
"evidence": ["p99_latency +340% on /checkout", "error class TimeoutError x1,204", "deploy event 7 min before onset"],
"proposed_action": "rollback payments to v3.2.0 (awaiting approval)"
}What can AI SRE agents do today?
Today's AI SRE agents are strong at evidence gathering and correlation across telemetry, useful for known failure patterns, and weak at novel failure modes and judgement calls. In production deployments the agent investigates and a human decides what to do with its findings.
| Capability | Maturity today |
|---|---|
| Evidence gathering across logs, metrics, traces | Strong: the core value; agents query faster and wider than a paged human |
| Correlating signals with deploys and config changes | Strong: mechanical joins over timestamps and metadata |
| Recognising known failure patterns | Good: improves with incident history and runbook context |
| Diagnosing novel failure modes | Weak: models lock onto one line of reasoning and miss multi-cause incidents |
| Judgement calls (blast radius, business trade-offs) | Human work: agents lack the context to weigh risk |
| Autonomous remediation | Not current best practice: propose-and-approve is the production norm |
In an August 2025 experiment, ClickHouse engineers injected four synthetic anomalies into an OpenTelemetry demo application and asked four frontier LLMs to find the root causes from telemetry alone. The authors concluded that autonomous RCA is not there yet: no model reliably identified causes without follow-up human guidance, and several fixated on a single early hypothesis even with full telemetry access.
AI SRE vs AIOps: what's the difference?
AIOps is the statistical layer: machine-learning models that detect anomalies, correlate related events, and reduce alert noise, established since Gartner coined the term in 2016. An AI SRE agent is the agentic layer, an LLM that takes a correlated incident and actively investigates it with follow-up queries. The agent typically consumes what AIOps produces.
| Classic AIOps | AI SRE agent | |
|---|---|---|
| Core technique | Statistical baselines, clustering, event correlation | LLM tool-use loop over telemetry APIs |
| Unit of work | Compress many alerts into one incident | Investigate one incident in depth |
| Output | A grouped, enriched incident record | Ranked hypotheses with query-backed evidence |
| Follow-up queries | None; humans run them | The defining behaviour of the agentic loop |
| Handles novel failures | Poor; knows only past patterns | Still weak, but can reason beyond training patterns |
| Era | ~2016 onward | ~2024 onward |
The two work as a pipeline: AIOps-style deduplication and correlation shrink the problem, then the agent digs into what remains.
Can AI replace site reliability engineers?
AI cannot replace site reliability engineers today. What AI SRE agents replace is investigation toil, the repetitive first hour of querying, correlating, and summarising. Removing that work compresses time-to-understanding and shifts human effort toward judgement about what a finding means, what action is safe, and what the business can tolerate.
"The AI hunts. The human decides." That framing, from Manveer Chawla's AI SRE architecture post, is the operating model that works in production. That division holds because the costs differ: a wrong hypothesis wastes minutes of agent time, while a bad autonomous fix can extend an outage. It also holds because reliability work is more than incident response: capacity planning, architecture review, defining error budgets and SLOs, and postmortem culture remain human-led. The realistic near-term change is fewer humans paged for evidence gathering, not fewer humans accountable for reliability.
Why does agent performance depend on the data layer?
An AI SRE agent is bounded by the queries it can run. An agent running 20 investigative queries against a store with 20-second latency needs nearly 7 minutes per hypothesis cycle, slower than a human with a dashboard.
Three constraints from ClickHouse's reference architecture determine what an agent can conclude:
- Retention. Platforms keeping 7-14 days of logs cannot support pattern-matching against last quarter's incidents or seasonal baselines.
- Cardinality. If user IDs, request IDs, and error codes were dropped to control cost, the agent cannot correlate an anomaly to a specific deployment, region, or customer.
- Query latency and cost. Multi-query loops collapse when each query takes tens of seconds or is billed per query.
Agents reach telemetry through APIs or a Model Context Protocol (MCP) server, which exposes the store's query interface as tools an LLM can call. Wix's deployment shows the ceiling when the data layer holds up, with unsampled, full-fidelity ingestion, hundreds of parallel agent queries, and 90% root-cause accuracy across 30,000+ monthly alerts.
Where ClickStack fits for AI SRE agents
ClickStack addresses the data-layer constraints directly. It stores logs, metrics, and traces as wide, high-cardinality events in ClickHouse, ingests OpenTelemetry natively, and serves the sub-second, high-concurrency SQL that multi-query agent loops require. Its agentic observability interface exposes semantic investigation tools to agents; in ClickHouse's internal evaluations it produced up to 25% fewer tool calls, 17% more accurate root-cause and remediation outcomes, and 2.7x more consistent investigations across runs. An agent's hypothesis check takes the same shape a human's would.
SELECT toStartOfMinute(timestamp) AS minute,
quantile(0.99)(duration_ms) AS p99
FROM traces
WHERE service = 'checkout'
AND timestamp > now() - INTERVAL 2 HOUR
GROUP BY minute
ORDER BY minuteHyperDX, the ClickStack UI, covers the human half of the loop with search, trace waterfalls, and dashboards over the same raw events, so approving or rejecting an agent's proposal means looking at the same data the agent queried. The agent's own behaviour needs monitoring too, but that is a separate discipline, AI agent observability, and the job of Langfuse, the open-source LLM observability platform that ClickHouse acquired in January 2026.
Frequently asked questions
How does an AI agent do root cause analysis?
An AI agent performs root cause analysis by iterating a query loop against the telemetry store, drafting candidate causes from evidence around the alert window and running targeted follow-up queries to confirm or eliminate each one. It reports ranked hypotheses with the supporting evidence; a human verifies them and decides on the fix.
How do I let an AI agent query my observability data?
Expose your telemetry store through an API or an MCP server with read-only credentials scoped to observability tables. The agent calls the query interface as a tool, so the store must handle many small analytical queries with low latency. Keep write and remediation permissions behind a separate human-approval path rather than granting them to the agent.
Can AI SRE agents fix incidents automatically?
Most production deployments keep remediation behind human approval: the agent proposes an action, such as a rollback, restart, or scale-out, and an engineer executes or confirms it. Autonomous execution is limited to well-understood, low-blast-radius runbook actions.
What data do AI SRE agents need?
Agents need raw, high-cardinality telemetry (logs, metrics, and traces with user IDs, request IDs, and error codes intact) plus deployment and config-change history for correlation. Retention should cover months, not days, so agents can compare against historical patterns. Pre-aggregated dashboards alone are insufficient, because investigation means asking questions nobody anticipated.
Building the data layer an AI SRE agent runs on? Try ClickStack, the open-source, OpenTelemetry-native observability stack built on ClickHouse.
