Isolating read and write workloads

​

Why isolate reads from writes

• Writes stop degrading reads. Continuous OpenTelemetry ingestion - the inserts themselves, plus the background merges that follow them - competes with dashboard and search queries for CPU and memory. Read latency can degrade noticeably while ingestion is running, and recovers once it stops.
• Reads stop disrupting writes. The contention runs both ways: a heavy ad-hoc query or an expensive dashboard render can exhaust memory on the service and fail inserts outright, not merely slow them down.
• Read-only compute is fully dedicated to queries. Read-only services perform no background merges outside of system tables. They also idle without delay, unlike read-write services, which merges can keep awake.
• Each side is sized on its own. The sizing model estimates ingest compute and query compute separately, and a warehouse lets you provision each as its own service. Above the model’s 1 QPS baseline query compute dominates - its worked example at 5 QPS arrives at 58 vCPUs for ingest against 290 for queries - so a small write service can feed a much larger read service.
• Idling and autoscaling are configured per service. Each service has its own replica count, autoscaling and auto-idling settings, so the write service can stay always-on for continuous ingestion while the read service idles outside working hours.
• Storage isn’t duplicated. Services in a warehouse share the same object storage folder and the same tables, and storage is billed only once.
• Access can be restricted per endpoint. IP access lists are applied per service, so the write endpoint can be reachable only from your collectors and the read endpoint only from your ClickStack deployment. See our guide on Network access control.

​

Architecture

• The first service in a warehouse is always read-write, and a service’s type is fixed at creation - to switch between read-only and read-write, create a new service in the warehouse.
• All services in a warehouse share the same cloud provider, region, ClickHouse version and Keeper, and the upgrade schedule of the primary service.
• Use one read-write service for ingestion. Merges are assigned across every read-write service sharing the storage, so a merge for an insert on one service can be executed by another. If that other service is also serving heavy queries, those queries compete with the merge for CPU and memory on the service running it - slowing merges for the first service’s inserts, and with them insert performance. Keep query workloads on the read-only service, and only add a second read-write service if you need to separate merges from ingestion.

​

Setting up an isolated deployment

CREATE DATABASE otel;
CREATE USER hyperdx_ingest IDENTIFIED WITH sha256_password BY '<strong-password>';
GRANT SELECT, INSERT, CREATE DATABASE, CREATE TABLE, CREATE VIEW ON otel.* TO hyperdx_ingest;

CLICKHOUSE_ENDPOINT=https://<read-write-service>.clickhouse.cloud:8443
CLICKHOUSE_USER=hyperdx_ingest
CLICKHOUSE_PASSWORD=<strong-password>
HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=otel

SELECT 
    user, 
    query_kind, 
    http_user_agent,
    count()
FROM clusterAllReplicas('default', system.query_log)
WHERE event_time > now() - toIntervalMinute(10) 
  AND type = 'QueryFinish'
  AND is_initial_query = 1
GROUP BY ALL
ORDER BY count() DESC;

SELECT 
    hostName() AS host, 
    query_kind, 
    count()
FROM clusterAllReplicas('all_groups.default', system.query_log)
WHERE event_time > now() - toIntervalMinute(10) 
  AND type = 'QueryFinish'
  AND is_initial_query = 1
GROUP BY ALL;

​

Separating merges from ingestion

• Don’t rely on auto-idling for either read-write service. A service with merges disabled still processes the part download and removal events generated by inserts elsewhere in the warehouse, and a high count of unmerged parts can block idling on its own. Plan for both read-write services to be continuously awake.
• Keep queries off both read-write services. Heavy SELECT queries on a read-write service compete with merge work for CPU and memory, which is the failure mode this topology exists to avoid. Point ClickStack at the read-only service as described above.
• Mutations, where you have them, are tracked on the service that executes them. Mutations are rare in observability - ClickStack’s schema sets ttl_only_drop_parts = 1, so ordinary retention drops whole expired parts during TTL merges rather than mutating rows away. If you do submit a mutation-producing ALTER to the ingest service, it is carried out by the merge service, and its progress appears in system.mutations there rather than on the ingest service.

​

Administration and DDL

• Table creation - performed automatically by the ClickStack collector on first ingest
• Modifying TTL to change retention
• Creating materialized views for query acceleration
• Adding skip indexes, projections and other performance optimizations

CREATE DATABASE otel
SETTINGS distributed_ddl_task_timeout=0

​

Isolating agentic workloads

claude mcp add --transport http clickstack https://mcp.clickhouse.cloud/clickstack \
  --header "x-service-id: <read-only-agent-service-id>"

​

Alerts

​

Isolating alert evaluation

• The common alerts, and any dashboards that go with them, exist only on the alerting service and aren’t visible to users working on the query service. Notifications are delivered to the same destinations either way, so what users lose is sight of the definitions, not the alerting itself.
• Sources on the alerting service are separate objects. Those using the default OpenTelemetry schema are auto-detected, but custom sources have to be configured there too before an alert can reference them.

​

Further considerations