48.9k
Log analytics is the practice of running analytical queries such as aggregations, group-bys, time-series rollups, and pattern detection over centralized log data to answer questions about system behavior, not just to find individual log lines. Where log search locates the one line that matters, log analytics computes error rates, latency percentiles, and traffic trends across billions of lines. Ride-hailing platform Didi runs this workload over petabytes of new log data every day.
Why does log analytics matter?
Log analytics matters because most production questions are aggregate questions. "Is the error rate rising?", "which endpoint got slower after the deploy?", and "which tenant is generating all the 429s?" can't be answered by reading log lines one at a time. They require computation over millions of lines at once.
This is also what separates analytics from log monitoring, which continuously watches for known conditions and alerts when they occur. Monitoring raises the alarm; analytics answers the ad-hoc questions that follow it.
Logs are also the highest-fidelity signal a system emits. Metrics pre-aggregate at collection time, which means they can only answer questions someone anticipated when the dashboard was built. Logs retain the raw detail, so new questions can be asked after the fact. That flexibility is what observability adds over plain monitoring. Google's SRE book frames log processing the same way: a retrospective source that trades immediacy for detail, used to reconstruct what actually happened.
How does log analytics work?
Log analytics works in three stages: collect, structure, query. Most of the value is created in the first two, since an unstructured pile of text supports search but not group-bys.
- Centralize. Agents and collectors ship logs from hosts, containers, and SDKs into one backend. This is the log aggregation step, typically handled today by the OpenTelemetry Collector or Fluent Bit.
- Structure. Fields like
service.name,severity, andduration_msbecome typed columns instead of substrings. The OpenTelemetry logs data model defines a standard record shape, and structured logging at the source is what makes it cheap. - Query. Aggregations, filters, and time-series rollups run over the structured store, increasingly in SQL, since analytical questions are naturally
GROUP BYquestions.
The surrounding lifecycle of retention, tiering, and archival is the domain of log management.
What is the difference between log aggregation and log analytics?
Log aggregation is the collection step: gathering logs from distributed sources into one queryable store. Log analytics is the consumption step: running analytical queries over what was aggregated. Aggregation without analytics gives you a centralized haystack; analytics is what turns the haystack into answers.
The two are often bundled in one product, which is why the terms blur. But they fail differently: an aggregation gap means data is missing entirely, while an analytics gap means the data exists and the questions still can't be answered, because the store only supports keyword search, or because aggregate queries over a few terabytes take minutes instead of seconds.
What are the challenges of log analytics at scale?
The core challenges of log analytics are volume, cardinality, and cost. Log data grows with traffic rather than with headcount or budget, analytical queries must scan far more of it than a point lookup does, and each challenge compounds the others as retention windows lengthen.
- Volume. A mid-size fleet produces terabytes per day; Didi reports petabytes daily. Every design decision downstream is a consequence of this number.
- High-cardinality fields. User IDs, container IDs, and trace IDs have millions of distinct values. Grouping by them is precisely what incident investigation needs, and precisely what many storage engines handle worst.
- Retention vs. cost. Analytical questions often reach back weeks ("was this slow before the last release?"), so logs must stay queryable, not just archived.
- Heterogeneous formats. JSON from one service, plain text from another, key=value pairs from a proxy: schema drift makes typed columns hard to maintain.
How do you query terabytes of logs quickly?
Fast queries over terabytes of logs come from matching the storage model to the query shape. Keyword lookups favor inverted indexes; aggregate queries favor columnar storage, which reads only the referenced columns and compresses repetitive log data well enough that scans touch a fraction of the raw bytes.
The two dominant storage models for logs make opposite bets:
| Dimension | Search-index model | Columnar-scan model |
|---|---|---|
| Core structure | Inverted index: token → matching documents | Columns stored contiguously, sorted and compressed |
| Built to answer | "Find the lines containing X" | "Compute Y across all lines matching a filter" |
| Aggregations and group-bys | Layered on top of the index, memory-heavy at high cardinality | The native operation: scan, group, reduce |
| Write cost | Tokenize and index every field at ingest | Append and compress; no per-token indexing |
| Storage footprint | Raw data plus index structures | Compressed columns, typically a fraction of raw size |
| Weak spot | Aggregate queries over long time ranges | Needle-in-haystack lookups without a filterable column |
Neither model is wrong; they optimize for different questions. Log analytics workloads (error rates over time, percentiles by endpoint, top-N by tenant) are scan-shaped, which is why analytical log stores are converging on columnar engines.
What does a log analytics query look like?
A log analytics query is an aggregate query with a time dimension: it filters a time window, groups by one or more fields, and computes a count, rate, or percentile per group. Error rate per service in five-minute buckets has this shape:
SELECT
toStartOfFiveMinutes(Timestamp) AS bucket,
ServiceName,
countIf(SeverityText = 'ERROR') / count() AS error_rate
FROM otel_logs
WHERE Timestamp > now() - INTERVAL 1 HOUR
GROUP BY bucket, ServiceName
ORDER BY bucket, error_rate DESCFinding the slowest endpoints from access logs is the same pattern with a percentile:
SELECT
LogAttributes['http.route'] AS endpoint,
quantile(0.95)(toFloat64OrZero(LogAttributes['duration_ms'])) AS p95_ms,
count() AS requests
FROM otel_logs
WHERE Timestamp > now() - INTERVAL 24 HOUR
GROUP BY endpoint
ORDER BY p95_ms DESC
LIMIT 10Both are illustrative schema shapes rather than tuned production queries, but they show the point: once logs are structured rows, log analytics is standard SQL, using the same aggregation skills a team already has.
What database is best for log analytics?
The best database for log analytics sustains high-rate ingest, compresses repetitive data well, aggregates quickly over high-cardinality fields, and keeps retention affordable at terabyte-to-petabyte scale. Those criteria describe a columnar analytical database more than a general-purpose one.
ClickHouse is an open-source columnar database built for exactly this query shape. Didi migrated its log platform to ClickHouse in 2024; the cluster spans more than 400 physical nodes, ingests over 40 GB/s at peak, and serves roughly 15 million queries per day. After moving off their previous search-index-based system, Didi's hardware costs fell about 30% and query speed improved roughly 4x. The criteria matter more than any vendor name: whatever the engine, petabyte-scale log analytics is won or lost on compression and scan speed.
Log analytics with ClickStack
ClickStack is the ClickHouse observability stack: ClickHouse as the analytical database, HyperDX as the UI, and a bundled OpenTelemetry collector for ingestion. Logs arrive over OTLP and land as structured wide rows in an otel_logs table, following the OpenTelemetry logs data model, so the SQL shapes above run against them directly and any log attribute is groupable without pre-declaring it.
That design addresses the challenges above directly: columnar compression keeps long retention affordable, and scan-oriented execution keeps high-cardinality group-bys fast at terabyte scale. HyperDX layers search, dashboards, and alerting over the same rows, so line-level lookups and aggregate analytics run against one store. The ClickStack documentation covers the schema and getting-started path.
FAQ
Do I need a separate tool for log analytics?
Not necessarily. Many teams start with the search features of their log aggregation stack and add a dedicated analytical store when aggregate queries become slow or expensive. That tipping point usually arrives with volume: a percentile query that takes seconds over gigabytes can take minutes over terabytes.
What is the difference between log analytics and log monitoring?
Log monitoring watches logs continuously for known conditions such as error spikes and failed health checks, and alerts when thresholds trip. Log analytics is the investigative side: ad-hoc aggregate queries asked after the fact, often questions nobody anticipated. In practice the two feed each other, since an alert from monitoring is usually the starting point for an analytical investigation.
Is log analytics the same as log aggregation?
No. Log aggregation centralizes logs from distributed sources into one store; log analytics queries that store analytically. Aggregation is a prerequisite, since you can't compute a fleet-wide error rate from logs scattered across hosts, but a system can aggregate perfectly and still lack the query engine that analytics requires.
Can you use SQL for log analytics?
Yes, and structured logs make it natural. Once log fields are typed columns, error rates become countIf aggregations, latency analysis becomes quantile functions, and time series become GROUP BY on a time bucket. SQL also lets logs join other data, such as spans sharing the same trace ID.
To run these query shapes over your own logs, try ClickStack, the open-source, OpenTelemetry-native observability stack built on ClickHouse.
