48.9k
An observability architecture is the end-to-end pipeline that moves telemetry from instrumented code to an engineer's screen: instrumentation, collection, transport, storage, query, and alerting. Most production designs standardize the first three layers on OpenTelemetry, the CNCF project second only to Kubernetes in contributor activity, which leaves storage and query as the layers where architectures genuinely differ.
TL;DR
- A reference observability architecture has six layers: instrumentation (OpenTelemetry SDKs), collection (OTel Collector), transport (OTLP), storage, query and visualization, and alerting.
- Four decisions shape everything else: agent vs gateway collectors, where sampling happens, how retention is tiered, and whether signals share one store or get one each.
- Telemetry outgrows the business data it describes: ClickHouse's internal logging platform accumulated 19 PiB of logs in six months, arriving at more than a million lines per second.
- Per-signal stores optimize each signal in isolation but push cross-signal correlation onto the engineer; a unified store turns correlation into a single query.
- ClickStack is a worked example of the unified design: a bundled OTel collector for ingestion, ClickHouse for storage and query, HyperDX as the UI.
What is observability architecture?
Observability architecture is the design of the system that produces, moves, stores, and queries telemetry: the logs, metrics, and traces an application emits. It spans six layers, from instrumentation SDKs inside application code to the dashboards and alert rules engineers consume, and its central decisions concern where data is aggregated, sampled, and stored.
Observability itself is a property: the ability to infer a system's internal state from its outputs. The architecture is what makes that property real, and it determines the two costs that matter most in practice: what you pay to keep telemetry, and how long an engineer waits for a question to be answered during an incident. The three pillars describe the data types flowing through the pipeline; the architecture describes the pipeline itself.
The term is also distinct from tool choice. Two teams running identical products can have very different architectures depending on where they place collectors, what they sample, and how long they retain data. The architecture is the set of those decisions, and swapping one vendor's dashboard for another changes it far less than moving sampling from the SDK to a gateway does.
What are the layers of an observability architecture?
An observability architecture has six layers, and OpenTelemetry standardizes the first three (instrumentation, collection, and transport). Storage, query, and alerting remain deployment choices.
application code + OTel SDKs 1. instrumentation
|
OTel Collector 2. collection
(receivers -> processors -> exporters)
|
OTLP over gRPC or HTTP 3. transport
|
telemetry store(s) 4. storage
|
dashboards, ad hoc query UI 5. query & visualization
|
alert rules -> paging/notification 6. alerting- Instrumentation. OpenTelemetry SDKs (stable for 11+ languages) generate spans, metrics, and log records inside the application, propagating request context via W3C Trace Context headers so one request stays joinable across services.
- Collection. The OpenTelemetry Collector is a pipeline of receivers (accept data in formats such as OTLP, Prometheus, or files via the filelog receiver), processors (batch, filter, sample, enrich), and exporters (send onward). Fluent Bit fills the same role in log-centric pipelines.
- Transport. OTLP, the OpenTelemetry Protocol, carries all three signals over gRPC or HTTP/protobuf, so one wire format serves the whole pipeline.
- Storage. Telemetry lands in one or more databases. This is the layer where architectures diverge most, covered in the decision sections below.
- Query and visualization. Dashboards render known questions; an ad hoc query interface answers unknown ones. Search-shaped and aggregation-shaped queries stress the storage engine differently.
- Alerting. Rules evaluate stored or in-flight telemetry and page someone. Google's SRE book chapter on monitoring distributed systems sets the bar: a page should be urgent, actionable, and require intelligence to resolve.
What are the key architectural decisions?
Four decisions define an observability architecture: collector topology, sampling location, retention tiering, and store layout. Each is a cost-versus-fidelity trade, and the storage choice constrains the other three.
| Decision | Options | Choose it when |
|---|---|---|
| Collector topology | Per-host agent / central gateway / agent + gateway | Agents alone for small fleets; add a gateway tier when you need fleet-wide sampling, redaction, or per-tenant routing in one place |
| Sampling location | Head-based (SDK decides at span start) / tail-based (gateway decides after the trace completes) | Head sampling is cheap and loses interesting traces; tail sampling keeps errors and slow requests but requires a gateway that buffers whole traces |
| Retention tiers | Single hot tier / hot + warm object storage / hot + archive | Tier when retention requirements (often 6-12 months for compliance) exceed what fast disks justify; object storage such as S3 carries the warm tier |
| Store layout | One store per signal / one unified store | Per-signal when teams already run mature single-signal systems; unified when cross-signal correlation and long retention drive the workload |
Several of these decisions become concrete in a gateway-tier collector configuration, abbreviated here:
receivers:
otlp:
protocols:
grpc:
http:
processors:
tail_sampling: # keep all errors, sample the rest
policies: [ ... ]
batch:
exporters:
otlphttp:
endpoint: https://telemetry-store.internal:4318Should you store telemetry in one database or one per signal?
Per-signal architectures give each signal a purpose-built backend: a search index for logs, a time-series database for metrics, a tracing backend for spans. A unified architecture lands all three signals in one store, typically a columnar database, as wide rows sharing trace and resource identifiers. The first optimizes each signal in isolation; the second optimizes the joins between them.
The per-signal design is the historical default for a reason. A metrics engine like Prometheus is exceptionally efficient at low-cardinality time series, and a log search index makes needle-in-haystack lookups fast. The costs surface during incidents: three query languages, three retention policies, and correlation done by a human copying timestamps and IDs between browser tabs. Log management alone becomes its own discipline per store.
The unified design keeps every signal queryable with one language and joins them on trace_id, so "show me the error logs for the slowest 1% of checkout traces" is a query rather than an investigation. That property matters more as AI SRE agents begin running investigations: ClickHouse's AI SRE observability architecture post shows an agent answering with a single SQL query that joins millions of log lines against deployment events, instead of making slow API calls to five separate tools. In exchange, the unified store must handle all three data shapes and high-cardinality aggregations at once, which is a demanding storage-engine requirement.
Hybrid layouts are common in practice. A team might keep Prometheus for infrastructure scrape metrics, where its pull model and alerting rules are entrenched, while landing logs and traces in a shared columnar store, then converge fully once the unified store proves it can serve the metrics workload too. The store-layout question tends to get settled gradually, starting wherever correlation hurts most.
Why is the storage and query layer the main cost?
Storage carries the bulk of the cost because telemetry outgrows the business data it describes: one user request commonly emits several log lines, a dozen metric samples, and a multi-span trace. ClickHouse's internal LogHouse platform ingests over a million log lines per second and stored 19 PiB across six months, compressed to 1.13 PiB, a 17x ratio.
Those numbers, reported in ClickHouse's 19 PiB logging platform write-up, make the architectural decisions concrete. Compression ratio decides whether six-month retention is affordable or absurd. Sampling location decides how much data reaches storage at all, and what gets lost on the way. Query speed decides whether long retention is useful; data that takes minutes to scan is archive, not observability. Whatever the store, the pattern that holds across production systems is that the instrumentation and transport layers are largely standardized and cheap, while storage-layer choices (engine, schema, compression, tiering) set both the bill and the debugging experience.
When should you redesign your observability architecture?
Redesign when incidents stall on cross-signal correlation, when retention is capped by cost rather than need, or when sampling is discarding the traces engineers ask for. If dashboards are merely ugly, change tools; if questions are unanswerable, change the architecture.
Three signals recur. First, engineers routinely export telemetry to spreadsheets or ad hoc scripts because the query layer cannot express the question; the storage engine is mismatched to the workload. Second, the retention window keeps shrinking to hold the bill flat, which means the compression and tiering layers are doing too little work. Third, on-call responders navigate three or more systems during a single incident, paying the per-signal correlation tax on every page. Any one of these justifies revisiting the decision table above before renewing a contract or scaling the existing design further.
What does an observability architecture look like in ClickStack?
ClickStack, the ClickHouse observability stack, implements the unified architecture end to end. A bundled OpenTelemetry collector handles the collection layer, ClickHouse stores and queries logs, metrics, and traces together, and HyperDX sits on top as the visualization and alerting UI.
Applications instrument with standard OTel SDKs and send OTLP to the ClickStack collector, which writes each signal into ClickHouse tables that keep full-resolution rows rather than pre-aggregates. Retention runs as table-level TTL policies with tiering to object storage, and HyperDX issues both its search and dashboard queries against the same tables. Cross-signal questions take the SQL shape the unified design promises (illustrative):
SELECT ServiceName, count() AS errors
FROM otel_logs
WHERE SeverityText = 'ERROR'
AND TraceId IN (
SELECT TraceId FROM otel_traces
WHERE Duration > 2_000_000_000 -- spans slower than 2s
)
GROUP BY ServiceName
ORDER BY errors DESCBecause every layer speaks OpenTelemetry, the architecture carries no proprietary agents: instrumentation written for ClickStack works unchanged against any OTLP-compatible backend.
FAQ
What is the difference between agent and gateway collectors?
An agent collector runs on every host or as a sidecar, close to the application, handling local enrichment and reliable forwarding. A gateway collector is a central, standalone tier that receives from many agents and applies fleet-wide policies: tail sampling, redaction, and routing. Production architectures commonly run both, agents feeding a gateway.
Where should sampling happen in an observability pipeline?
Head-based sampling happens in the SDK when a trace starts: cheap, but it discards traces before knowing if they are interesting. Tail-based sampling happens in a gateway collector after a trace completes, keeping errors and slow requests while dropping routine traffic. Teams that alert on rare failures generally need tail sampling despite its buffering cost.
Do you need a separate database for logs, metrics, and traces?
No. Per-signal stores are common because specialized engines predate unified ones, but all three signals can live in one columnar store as wide rows sharing trace identifiers. The deciding factor is workload: isolated dashboards favor specialized stores, while incident investigation and long retention favor one store where signals join in a single query.
What is OTLP and why does it matter?
OTLP, the OpenTelemetry Protocol, is the wire format that carries logs, metrics, and traces between SDKs, collectors, and backends, over gRPC or HTTP with protobuf encoding. It matters because one protocol for all three signals removes per-signal shippers and format converters, and any OTLP-compatible backend can receive the same instrumentation unchanged.
How do you reduce telemetry data costs?
Four levers, in order of leverage: sample traces at the gateway so routine traffic never reaches storage, structure events at the source so they compress well, use columnar storage (ClickHouse measured 17x compression on its own logs), and tier retention so older data moves to object storage instead of fast disks.
To see the six layers running as one stack, try ClickStack, the open-source, OpenTelemetry-native observability stack built on ClickHouse.
