ClickHouse gives users an incredible amount of control to optimize database performance. Platform administrators sit at the center of this experience: they tune scaling controls, configure server settings, and ultimately own the health of their deployment. On ClickHouse Cloud, we're releasing a set of improvements designed to give deeper visibility into how the ClickHouse server is behaving to proactively surface warnings before they become problems.

<h2 id="more-dashboards">More dashboards</h2>

The ClickHouse Cloud console already comes built-in with several monitoring dashboards. We added a few more.

<h3 id="new-overview-page">New overview page</h3>

The new Overview page brings the most important signals about your deployment into a single, unified view. Designed to give administrators an at-a-glance health check of their ClickHouse environment, you can now spend less time hunting for information and more time acting on it.

<h3 id="infrastructure-page-deeper-scaling-visibility">Infrastructure page: Deeper scaling visibility</h3>

The new Infrastructure page gives administrators a clearer view into how their services are scaling over time. CPU and Memory utilization metrics, now with additional aggregation types, show how much utilization your ClickHouse cluster is getting.

New modals showcase ClickHouse's [automatic scaling](https://clickhouse.com/docs/manage/scaling) behavior to help users better understand why and how their cluster scaled, and also make better decisions around custom scaling configurations like vertical scaling limits and idling behavior.

![](/uploads/clicklens_mar2026_image1_621e99285e.png)

<h2 id="get-ahead-of-issues-with-new-notifications">Get ahead of issues with new notifications</h2>

When users onboard onto ClickHouse, a common set of issues frequently come up. Administrators will be automatically notified via email when a service is at risk of degraded performance or failures, including due to:

* **Too many parts**: a common cause of merge pressure and query slowdowns
* **Failed mutations**: which can silently stall data changes if left undetected
* **Query concurrency**: helping you catch saturation before high query concurrency impacts end users

<h3 id="slack-notifications">Slack notifications</h3>

You can configure all these notifications, and more, to be sent directly to Slack via your organization's notification settings.

These improvements provide administrators with better visibility into ClickHouse behavior, so you can spend less time diagnosing issues and more time optimizing your performance.

These features are being rolled out now in ClickHouse Cloud. Log in to your console to explore the new dashboards and configure your notification preferences.


Interested in seeing how ClickHouse works on your data? Get started with ClickHouse Cloud in minutes and receive $300 in free credits.

Get started today

 ClickHouse Cloud's new monitoring capabilities give platform administrators deeper visibility into their deployments, with a unified   Overview page, infrastructure scaling insights, and automatic email and Slack notifications for common issues.

large_clicklens_mar2026_image2.png

medium_clicklens_mar2026_image2.png

small_clicklens_mar2026_image2.png

thumbnail_clicklens_mar2026_image2.png

clicklens_mar2026_image2.png

More Visibility, Less Guesswork: ClickHouse Cloud's New Monitoring Capabilities

ClickHouse Cloud Dashboards are now Generally Available. Last December, we [announced the beta release of Dashboards](https://clickhouse.com/blog/reinvent-2024-product-announcements#dashboards-beta) in the ClickHouse Cloud console, a feature that allows users to create visual dashboards, collect insights, and share visualizations from saved queries. Over the six months following launch, we've been actively collecting user feedback, analyzing feature adoption, and shipping additional features like sharing.

In this blog post, I'll share how we used our own dashboards feature to track the feature's adoption and outline our vision for the future of ClickHouse Cloud Dashboards.

## The problem we solved

ClickHouse is an exceptional database for analytics, and ClickHouse Cloud provides the best way to deploy it. While the ClickHouse Cloud console has featured a [SQL console](https://clickhouse.com/docs/integrations/sql-clients/sql-console) for several years, it had one significant limitation: users needed to write and execute SQL queries to access their data.

This created a barrier for wider adoption within organizations. Internally at ClickHouse, we use ClickHouse Cloud as our data warehouse, but many team members, particularly in non-technical roles, didn't want to write SQL or even see SQL code to view data. To address this, we deployed [Superset](https://superset.apache.org/) on top of our ClickHouse Cloud service and created dashboards for the team's consumption.

The core issues were clear: many users aren't comfortable writing SQL and expect dashboard-based data access. Additionally, our SQL console made it impossible to view results from multiple queries side-by-side.

## How Dashboards work

At their foundation, dashboards are visual representations of `SELECT` queries, we designed our dashboards feature with this principle at its core. To [create a dashboard](https://clickhouse.com/docs/cloud/manage/dashboards#create-a-dashboard), users simply `SELECT` data from a table and save the query. Through Dashboards, users can select saved queries (or write new ones) and create visualizations from these queries.

![dashboard-screen.png](https://cms.clickhouse-dev.com:1337/uploads/dashboard_screen_61497407be.png)

<p style="text-align: center; font-style: italic;">
  A dashboard powered by the "Events over time" query
</p>

Dashboards leverage [Vizhouse](https://github.com/ClickHouse/viz-house), the charting library powering all ClickHouse Cloud frontend applications. Vizhouse supports multiple visualization types including tables, bar charts, line charts, pie charts, and more. We also added helper elements like spacer bars and text boxes to help users organize and annotate their dashboards.

Using ClickHouse Query Parameters, dashboards can be made interactive. In addition, after significant user demand, we introduced three permission levels, allowing dashboard creators to share their work with view-only users.

:::global-blog-cta:::

## Tracking launch and adoption

When we launched Dashboards, we prioritized detailed telemetry to understand user behavior. We accomplished this with just two tables.

We use ClickHouse to store frontend logs, capturing clicks, query runs, dashboard creations, visualization creations, and more. All events are stored in a single ClickHouse table called `forensics_v2`:

<pre><code type='click-ui' language='sql'>
CREATE TABLE forensics_v2
(
    `created_at` DateTime('UTC') DEFAULT now(),
    `environment` LowCardinality(String),
    `session_id` Nullable(String),
    `request_id` Nullable(String),
    `server_ip` Nullable(IPv4),
    `org_id` Nullable(UUID),
    `user_id` Nullable(String),
    `namespace` Nullable(String),
    `component` Nullable(String),
    `event` String,
    `interaction` LowCardinality(String),
    `payload` Nullable(String),
    `message` Nullable(String)
)
ORDER BY created_at;
</code></pre>

A separate table stores demographic information about each organization, which we join to the events table for enrichment with customer names, cohorts, and other attributes. Our product analytics for this feature launch relied primarily on these two tables.

The scale of our analysis demonstrates ClickHouse's power. Our query tracking dashboard query scans 144 million rows in under 1.5 seconds. The only ETL required was importing frontend logs into ClickHouse – all further processing of the data was accomplished via simple `SELECT` queries that are run every time the dashboard is loaded.

<pre><code type='click-ui' language='sql'>
SELECT uniq(user_id)
FROM forensics_v2
WHERE namespace = 'dashboards'
AND component = 'general'
AND event = 'createDashboard';
</code></pre>

<p style="text-align: center; font-style: italic;">
Query to calculate total number of users who created a dashboard.
Read: 18,992,795,602 rows (254.48 GB); Elapsed: 3.748s. 
</p>

We identified a couple key performance indicators to track, including the number of users creating dashboards, users viewing dashboards, and query runs. We defined these KPIs using SQL and saved them as queries to support our data model.

With our queries in place, I created a dashboard to visualize these KPIs and identify our most active dashboard users. This data helped us prioritize our roadmap and collect targeted feedback by proactively reaching out to power users.

## Common use cases

During our six-month beta period, three primary use cases emerged:

- **Simple Visualizations**: Dashboards enable users to create straightforward data views, often with just one or two visualizations. Users found dashboards provided easier data access compared to running queries in the SQL console.

- **Monitoring**: ClickHouse's system tables are powerful monitoring tools. Many users created queries against these system tables to [build monitoring dashboards](https://clickhouse.com/blog/essential-monitoring-queries-creating-a-dashboard-in-clickHouse-cloud) for their ClickHouse services, supplementing existing monitoring tools.

![dashboard-gif.gif](https://cms.clickhouse-dev.com:1337/uploads/dashboard_gif_24b25599e2.gif)

- **Feature Usage Analytics**: Similar to our own adoption analysis, many users leveraged Dashboards for product analytics on logs and events data. Users who were comfortable writing SQL found it easier to share insights with teammates who were less comfortable with SQL.

## Future roadmap

Looking ahead, our dashboard roadmap focuses on three key areas:

- **Enhanced Visualizations**: We're improving the core dashboard experience by adding visualization options like dimensions and expanding chart types.

- **AI Integration**: Given AI's proficiency with SQL and the existing possibility of using MCP with ClickHouse, we plan to integrate an intelligent "business analyst" capability that can create and analyze dashboards within the SQL console.

- **Embeddable/Public Dashboards**: We're developing functionality to allow users to embed dashboards directly into their applications.




ClickHouse Cloud Dashboards are now GA! See how we use them to track adoption, gather insights, and shape what’s next.

large_02 - Tracking the adoption of a new feature.png

medium_02 - Tracking the adoption of a new feature.png

small_02 - Tracking the adoption of a new feature.png

thumbnail_02 - Tracking the adoption of a new feature.png

02 - Tracking the adoption of a new feature.png

ClickHouse Cloud Dashboards: Tracking the adoption of a new feature

Monitoring ClickHouse can feel scattered, especially when jumping between system tables and ad hoc queries. In a [previous blog post](https://clickhouse.com/blog/monitoring-troubleshooting-select-queries-clickhouse), the ClickHouse Support team provided some “essential monitoring queries” to help users monitor their ClickHouse instance using ClickHouse’s powerful system tables. In this post, we’ll turn those essential queries from our earlier blog into a centralized, reusable dashboard in ClickHouse Cloud, giving you real-time visibility in just a few clicks.

Here’s the end result of what we’ll create:

![Untitled2.gif](https://cms.clickhouse-dev.com:1337/uploads/Untitled2_daa070c137.gif)

We’ll start by running a query on `system.part`s to provide a global overview of my cluster. Specifically, it shows the biggest tables in terms of rows, data and primary key size:

<pre>
<code type='click-ui' language='sql'>
SELECT
    table,
    sum(rows) AS rows,
    max(modification_time) AS latest_modification,
    formatReadableSize(sum(bytes)) AS data_size,
    formatReadableSize(sum(primary_key_bytes_in_memory)) AS primary_keys_size,
    any(engine) AS engine,
    sum(bytes) AS bytes_size
FROM clusterAllReplicas(default, system.parts)
WHERE active
GROUP BY
    database,
    table
ORDER BY bytes_size DESC
</code>
</pre>

In ClickHouse Cloud, we’ll save this query, and add it to a new dashboard - `Monitoring SELECT queries`:

![unnamed (1).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_1_efc260a1d6.png)

![unnamed (2).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_2_e88c2e9e9f.png)

When I navigate to the Dashboards tab, I’ll now see a brand new dashboard with a table component that I just added:

![unnamed (3).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_3_62ac8852c9.png)

I’ll keep adding a few more queries from the blog post to this dashboard as tables. I’ll create my first line chart showing average query duration and number of requests, which works great as a time-series line graph in the SQL console:

![unnamed (4).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_4_50b86a0016.png)

I’ll add this visualization to our dashboard. I’ll also add a legend and change the formatting of my series so hovering over the series only shows one decimal point:

![unnamed (5).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_5_5c0d199cea.png)

Finally, I’ll add one more table showing which users have been running the most queries recently. I now have a dashboard which I can keep reference to monitor queries on my cluster. As a final step, I’ll use ClickHouse [Query Parameters](https://clickhouse.com/docs/sql-reference/syntax#defining-and-using-query-parameters) to add a time-based filter to my line chart so the dashboard is interactive. I’ll modify the underlying query, and save it:

![unnamed (6).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_6_a99e3a5922.png)

I’ll configure a "Filter" as the value source for this filter, and keep a 3-day default:

![unnamed (7).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_7_726a732d61.png)

Now, viewers of my dashboard can toggle how many days of data they see on the line graph:

![unnamed (8).png](https://cms.clickhouse-dev.com:1337/uploads/unnamed_8_c76c2a1edf.png)

Here’s the final result! Each dashboard has a custom URL, so it’s easy to share this dashboard with colleagues in the same Cloud organization via the browser’s URL:

![Untitled2.gif](https://cms.clickhouse-dev.com:1337/uploads/Untitled2_daa070c137.gif)

With this dashboard in place, you’ll spend less time hunting down queries you previously wrote, and more time optimizing performance. Try building your own with sample queries, or customize queries to fit your specific workload.

A short, easy-to-follow guide to building your own monitoring dashboard in ClickHouse Cloud. Real-time visibility. Zero setup. Just a few clicks.

large_Screenshot 2025-05-20 at 13.02.19.png

medium_Screenshot 2025-05-20 at 13.02.19.png

small_Screenshot 2025-05-20 at 13.02.19.png

thumbnail_Screenshot 2025-05-20 at 13.02.19.png

Screenshot 2025-05-20 at 13.02.19.png

Essential Monitoring Queries: Creating a Dashboard in ClickHouse Cloud

![whast_new_2025.png](/uploads/whast_new_2025_67023db754.png)

If you live in the northern hemisphere, spring is here! As flowers bloom and animals come out of hibernation, we’ve been bringing the power of ClickHouse to ClickHouse Cloud. Over the past few months, we’ve delivered a wave of new features and improvements — all designed to give you the most powerful backend for your real-time data as you spring into the new season. Here’s a look at what’s new from the [ClickHouse Cloud Changelog](https://clickhouse.com/docs/whats-new/changelog/cloud).

## Bring Your Own Cloud (BYOC) - now Generally Available for AWS

In February, we [introduced a new deployment model](https://clickhouse.com/blog/announcing-general-availability-of-clickhouse-bring-your-own-cloud-on-aws) for ClickHouse by launching Bring Your Own Cloud (BYOC) in GA for AWS. With BYOC, you can deploy ClickHouse Cloud directly into your own AWS account, maintaining full control of your data while we manage the operations. It’s the best of both worlds: the power of ClickHouse with the security and compliance benefits of a single-tenant environment.

![byoc.png](/uploads/byoc_5942e30172.png)

Learn more about the architecture, onboarding process, and operations about our BYOC offering [here](https://clickhouse.com/blog/building-clickhouse-byoc-on-aws). To request access or add yourself to the waitlist for GCP or Azure, please submit your information [here](https://clickhouse.com/cloud/bring-your-own-cloud). 

## Native Postgres Change Data Capture (CDC) - in public beta

After [joining forces with PeerDB](https://clickhouse.com/blog/clickhouse-acquires-peerdb-to-boost-real-time-analytics-with-postgres-cdc-integration), in February we announced the public beta of a native Postgres CDC connector, making it easier than ever to capture and replicate Postgres changes into ClickHouse in near real-time. 

<blockquote>
<p>“ClickPipes for Postgres has made it incredibly easy for us to keep our billing data in Postgres synchronized with ClickHouse for efficient analytics. The CDC experience is blazing fast, ensuring data freshness within seconds while minimizing the load on our production Postgres database. An invaluable solution for seamlessly integrating Postgres with ClickHouse!”</p><p>Mo Abedi, Software Engineer in Billing team, Neon.tech</p>
</blockquote>

Learn more about this feature in our [launch blog](https://clickhouse.com/blog/postgres-cdc-connector-clickpipes-public-beta). During the public beta, if you face issues, have questions, or want to chat with the team working on this feature, please reach out to [db-integrations-support@clickhouse.com](mailto:db-integrations-support@clickhouse.com).

## Native MySQL Change Data Capture (CDC) - in private preview

When we launched the Postgres CDC connector, we got overwhelming requests for a similar MySQL connector - so we developed a ClickHouse native CDC connector purpose built for MySQL which is now in private preview. You can use this connector for both continuous replication and one-time migration from MySQL, no matter where it's running—whether in the cloud (RDS, Aurora, CloudSQL, Azure, etc.) or on-premises.

Some key features of this connector include blazing fast backfills during the initial load, continuous replication from MySQL, schema change replication, and more. 

Learn more in our [launch blog](https://clickhouse.com/blog/mysql-cdc-connector-clickpipes-private-preview), and sign up for the Private Preview [here](https://clickhouse.com/cloud/clickpipes/mysql-cdc-connector).

## Slack notifications for Cloud events

Monitor your ClickHouse Cloud deployment without leaving Slack. ClickHouse Cloud already sends notifications for key events — such as billing, scaling, and ClickPipes. In April, we started allowing users to deliver these notifications directly to your Slack workspace using the ClickHouse Cloud Slack application. 

![alert.png](/uploads/alert_b8b6678f95.png)

To get started, admins can configure these notifications via the notification center by specifying slack channels to which notifications should be sent. Learn more [here](https://clickhouse.com/docs/cloud/notifications). 

## Monitor ClickHouse with the Resource Utilization Dashboard

Also in April, we started rolling out a new dashboard to monitor your ClickHouse Cloud deployment without leaving the ClickHouse Cloud console. The new Resource Utilization dashboard gives replica-level insights into how your cluster is sized, also how much CPU and memory load each replica is experiencing, and how much data is transferred in and out of your service. 

We scrape the metrics on this dashboard from [ClickHouse system tables](https://clickhouse.com/docs/operations/system-tables), and serve them via this dashboard to help you diagnose issues of overprovisioned or underprovisioned clusters. For one of our internal ClickHouse Cloud services, we used this dashboard to cut our ClickHouse Cloud costs by 4x! 

Questions, comments, or feedback? Reach out to [metrics-requests@clickhouse.com](mailto:metrics-requests@clickhouse.com). 

![resource_util.png](/uploads/resource_util_dc54995e2e.png)

## New region: AWS Middle East (UAE) - me-central-1

In February, we’re announced that ClickHouse Cloud is now available in the AWS Middle East (UAE) region (me-central-1). This expansion helps us better serve customers in the Middle East who require local data residency and low-latency access. With this expansion, you can now harness the power of ClickHouse in [11 regions on AWS, 4 regions on GCP, and 3 regions on Azure](https://clickhouse.com/docs/cloud/reference/supported-regions). We also support private regions in select geographies, you can find more information or request access [here](https://clickhouse.com/docs/cloud/reference/supported-regions).

## Cross-Region Private Link - in public beta

In March, we announced the public Beta of Cross-Region Private Link - useful for customers running ClickHouse clusters across multiple AWS regions. This enables secure access between your applications and ClickHouse Cloud, no matter where they live.

Learn more [here](https://clickhouse.com/docs/manage/security/aws-privatelink).

## ClickPipes - AWS PrivateLink

Last week, we released the ability to use AWS PrivateLink and ClickPipes to establish secure connectivity between VPCs, AWS services, your on-premises systems, and ClickHouse Cloud. This can be done without exposing traffic to the public internet while moving data from sources like Postgres, MySQL, and MSK on AWS. It also supports cross-region access through VPC service endpoints. PrivateLink connectivity set-up is now fully self-serve through ClickPipes.

Learn more [here](https://clickhouse.com/docs/integrations/clickpipes/aws-privatelink).

## Usage cost API endpoint

A common ask from users was better monitoring around billing and costs, so in February we started exposing a new API endpoint to allow you to programmatically retrieve usage and cost data for your ClickHouse Cloud organization. Whether you’re building internal dashboards or automating budget tracking, it’s easier than ever to stay on top of your ClickHouse Cloud spend.

![usage_api.png](/uploads/usage_api_225d68d4d9.png)

*Vantage Usage Report showing ClickHouse Cloud usage and costs, powered by the Usage Cost API under the hood.*

## New user roles

We expanded our role-based access control to give teams more flexibility with access control.
* New Member Role (Organization-Level): Member is an organization level role that is assigned to SAML SSO users by default and provides sign-in and profile update capabilities.
* Service-Level Roles: We also introduced two new roles that can be scoped to services:
    * Service Admin: Full control of assigned services.
    * Service Read Only: View-only access. 

These service roles can be assigned to users who already have a Member, Developer, or Billing Admin role. Refer to [Access control in ClickHouse Cloud](https://clickhouse.com/docs/cloud/security/cloud-access-management/overview) for more information.

## PCI and HIPAA compliance

Security and compliance are at the core of everything we do. In addition to an already exhaustive list of [security and compliance reports](https://clickhouse.com/docs/cloud/security/security-and-compliance), ClickHouse Cloud began to meet key requirements for PCI DSS and HIPAA in February. If you need to build analytics applications for regulated industries like healthcare and finance, we have you covered. We support PCI and HIPAA in select regions at this time, please refer to the documentation for the list of [supported regions](https://clickhouse.com/docs/cloud/reference/supported-regions#hipaa-compliant-regions). To request access, please review guidelines [here](https://clickhouse.com/docs/cloud/security/security-and-compliance).

## We’re Just getting started: summer is coming

These updates are part of our commitment to delivering the most powerful, flexible, and user-friendly ClickHouse backend for your application. As always, we'd love to hear your feedback. You can view the full changelog [here](https://clickhouse.com/docs/whats-new/changelog/cloud), engage with our team directly in our [community Slack](https://clickhousedb.slack.com/ssb/redirect), and [get started with a free trial](https://console.clickhouse.cloud/signUp).

Stay tuned for more.



Spring 2025 brings major updates to ClickHouse Cloud - from BYOC and native CDC connectors to Slack alerts, usage APIs, and expanded regional support.

large_whast_new_2025.png

medium_whast_new_2025.png

small_whast_new_2025.png

thumbnail_whast_new_2025.png

whast_new_2025.png

What's new in ClickHouse Cloud: spring 2025 roundup


A year ago, my colleague Dmitry Pavlov described [how we built ClickHouse’s internal data warehouse on ClickHouse Cloud](https://clickhouse.com/blog/building-a-data-warehouse-with-clickhouse), nicknamed “DWH.” This DWH serves internal business analytics and reporting at ClickHouse, Inc. by collecting data from various source systems, processing it according to a data model, and providing access to the data. The original blog post covers several aspects of building DWH like the use of denormalized data marts, implementation of security features, management of the underlying infrastructure, and more.

However, as ClickHouse has grown, its ClickHouse Cloud based DWH has grown with it. In this blog post, I’ll share an update on how the DWH has evolved over the past year to support a more diverse set of users, data sources, and access points. In particular, our continued use of ClickHouse and the incorporation of [dbt](https://www.getdbt.com/product/what-is-dbt) as two primary components in the stack have enabled DWH to support more real-time data processing into regular batch reporting, a feature that other traditional data warehouses have historically lacked. As a result, we’re able to make better decisions from our data. 

## What is a data warehouse?

A data warehouse is a centralized repository that stores data collected from disparate sources within an organization to support decision making, typically business decisions. Here’s an overview of our DWH’s architecture. 

![01_unnamed (4).png](/uploads/01_unnamed_4_e8a54ce5b4.png)

At present, we have configured nineteen raw data sources with either hourly or daily insert jobs. Uncompressed, around 6 billion rows and 50 TBs of data are written to DWH every day. ClickHouse then compresses and stores this data. At present, we’re storing 470 TBs of data (compressed) collected over the last two years, which has increased 23% over the last 30 days. In a given week, we typically see ~70 unique users who generate at least one SELECT query.

DWH is a business-critical system in our organization because of the various business units that depend on DWH for data including leadership, product, engineering, marketing, sales, and finance. In addition, the DWH pre-production environment has been used as a canary to test open-source ClickHouse releases before performing broader upgrades in ClickHouse Cloud. 

## Setting up batch reporting

The first iteration of our DWH was as a tool to import data from a small number of business systems with relatively small data volumes. After importing this raw data, it was processed to create batch reports like “# of trials” or “conversions,” which are the foundation of a data warehouse. Batch reporting was largely configured in [Airflow](https://airflow.apache.org/), and we didn’t use dbt at all. However, as we added data sources, defined more complex business metrics, and served more and more internal stakeholders, this approach did not scale. dbt is an open-source tool used by data engineers and analysts to build data pipelines, allowing teams to write and iterate on modular SQL code used to transform data. Introducing dbt to the stack helped because it centralized transformation logic related to batch reporting in one place.

For example, when we add new data sources, dbt allows us to use time functions to easily join data that is supplied at different times, triggering follow-on processes when ready. For example, consider the scenario where data from different sources is inserted at different times, say 7:26 pm and 10:01 pm, even though the logical timestamp used to JOIN data is one hourly increment, like 7:00pm. This logic can be highly custom, and is subject to change based on business needs. Encoding this logic with 19 different data sources is a lot easier when there’s one tool like dbt to track dependencies which also automatically produces lineage graphs as documentation. 

In addition, the metrics we defined became more complicated. Simple count()s and sum()s across a date range gave way to window functions and data differencing which filtered on conditions stored in external tables. dbt paired with ClickHouse’s SQL dialect was key to managing this growing complexity as SQL became the way we encoded this business logic. dbt became the foundation of a more scalable way to report batch-processed business metrics in DWH.


## Incorporating more real-time data

The ability to define complex business metrics as part of batch reporting processes is table stakes for an enterprise data warehouse. ClickHouse, known for its performance over petabyte-scale datasets, easily handles the data that we throw in. However, where our DWH really excels is in its ability to handle new and unique real-time data formats that don’t conform to strict schemas or relations (support for which is constantly [improving](https://github.com/ClickHouse/ClickHouse/issues/58392)). Often, these data sources also have much bigger data volumes.

For example, in our DWH, every day, ~4 billion rows are inserted into just one of the tables - the entire table has close to a trillion rows. To give another example, we import events from Google Analytics which stores event parameters as nested arrays that represent website activity on clickhouse.com. Similarly, offloads from our NoSQL Control Plane database resemble a nested JSON structure with a flexible schema.

With our foundational batch reporting in place, we began to configure more and more real-time data sources into our reporting. Our users shared feedback that they found this kind of data to be very valuable and more intuitive, even if the data was less structured and required some light processing when running queries (for example, extracting fields from JSON columns). Our Marketing team was able to get a better sense of which pages on our website were most popular. Our Support Engineers were able to diagnose problems with customers’ queries by digging into logs of their recent Cloud activity. Our product team was able to define more custom logic to track conversion events. 

![02_unnamed (4).png](/uploads/02_unnamed_4_aaf41defa7.png)

In DWH, we exposed this real-time data both in its raw format as we imported it, as well as in a transformed state, like aggregations of real-time events. [ClickHouse’s rich library of functions](https://clickhouse.com/docs/en/sql-reference/functions) and supported data formats made exploring real-time data in its raw format very easy, meaning users were able to perform ad-hoc analysis with a SQL client without needing help from data engineers. Creating aggregations of real-time data was also possible - we used dbt to define aggregations as real-time data arrived in DWH, and then stored these aggregates in a separate table. We used dbt because we were already using dbt elsewhere so it was easier to centralize in an existing tool, but these aggregations are also natively configurable in ClickHouse using features like [materialized views](https://clickhouse.com/docs/en/materialized-view). These aggregations are then joined with existing reporting to track metrics like “number of customers with failed queries.” 

This setup allows us to configure batch-reporting style pipelines for real-time data while preserving the underlying records in the event that deeper investigation is needed. In one environment, ClickHouse Cloud + dbt lets our users combine batch-processed reporting with real-time data streams.

## Configuring additional access points

In the first iteration of DWH, we configured [Apache Superset](https://superset.apache.org/) as the BI tool for users to access data. Superset provides a number of key features like dashboarding, a SQL client, alerting, and user management in one deployment which was attractive to the team maintaining DWH. Also, it’s open-source software. However, it had its limitations. Namely, we found Superset’s SQL client to be buggy, which has a very real impact on user experience.

To address this issue, we opened access to DWH using ClickHouse Cloud’s native SQL console. The SQL console was far better for users writing ad-hoc SQL queries, and for exploring the various database tables and views. We use the ClickHouse Cloud API to manage user access to sensitive data, and use Google Groups to assign roles to users. Several users of DWH shared how using the SQL console was a superior experience to Superset’s native SQL client. 

For a more specialized use case beyond writing ad-hoc queries, we configured a connection with [Growthbook](https://www.growthbook.io/) as a way to run A/B tests using data in our DWH. The connection between Growthbook and ClickHouse Cloud takes a matter of minutes to set up, and provides us with an out-of-the-box tool for analyzing results of A/B tests (which are SELECT queries to the DWH). Growthbook queries the DWH ClickHouse Cloud service directly, which means that running experiments using very raw, log-level data is surprisingly trivial. 

Finally, we set up a data exporter job from ClickHouse Cloud to Salesforce so that our sales team can consume data from DWH directly in Salesforce, our CRM. This connection is a bit trickier to set up because our Salesforce deployment doesn’t support queries from a static IP address, a limitation that would have made the connection less secure. In the end, we ended up pushing data from DWH into an S3 bucket, and then letting Salesforce query this S3 bucket instead. 

## Where we want to take DWH next

Looking to the future, we’re excited to keep scaling DWH by configuring additional data sources and onboarding more users. To ensure DWH scales, we want to decompose workloads to independently scalable compute groups which we believe will lead to performance improvements and cost savings.

When DWH was first created, we had just one “hourly load DAG” query which was one ETL job scheduled in Airflow that was responsible for incorporating all new data into DWH. This architecture was chosen because we had a limited number of data sources with a simple data model comprising just two entities. However, we quickly realized that this approach was not scalable as we added additional data sources and data models: The insert job was taking too long to execute, and just one error would jeopardize insertion of all data because of dependencies among different data models. After introducing dbt, we refactored this one job into nine different processes and noticed improvements in service reliability. Most of these jobs still run every hour, meaning that our ClickHouse Cloud service has to be running 24x7 to support insertion of new data.

An upcoming ClickHouse Cloud feature that we’re excited for is [compute-compute separation](https://clickhouse.com/docs/en/cloud/reference/compute-compute-separation). This feature lets us separate DWH’s compute resources into multiple services, each of which can scale up or scale down independently as needed. This means we can have a read-only service for our BI tool, a 24x7 read-write service for critical ETL jobs, and a second read-write service for less critical ETL jobs where data only needs to be refreshed once a day. This second read-write service can then be scaled down when inactive, yielding considerable cost savings. In this manner, we see potential to decentralize compute resources and make different compute groups work independently of one another. 

In the age of AI, I’m also excited to see how we can incorporate AI features into our DWH. ClickHouse Cloud already supports an AI query builder that assists users in writing simple SQL queries. However, in the long term, I’m excited by the prospect of an AI business analyst who can assist users of our DWH that aren’t intimately familiar with SQL. Also, a number of DWH users currently query the DWH to write weekly/monthly/quarterly reports, another process that I’d love to task to an AI by using existing reports and queries to provide context. 

## Closing thoughts

At ClickHouse, we begin each week with a leadership meeting to review and discuss the most important metrics for the company. Many of these metrics are reported directly from the DWH. As our organization has scaled, and DWH with it, I’ve realized that data warehouses serve an important role beyond just displaying pretty dashboards and generating reports; Data warehouses are complex, powerful machines that facilitate discussion and spur action by aligning entire organizations of people to a single source of truth. They allow engineers, marketers, sellers, product managers, and executives to consume the same information and make decisions accordingly. For our data warehouse, ClickHouse expands the universe of queryable data by easily enriching batch reports with real-time data.

Our own internal data warehouse with ClickHouse and only open-source technology. Let's find out how things are going, a couple of years after we built it.

Mihir Gokhale

Blog posts by Mihir Gokhale

More Visibility, Less Guesswork: ClickHouse Cloud's New Monitoring Capabilities

ClickHouse Cloud Dashboards: Tracking the adoption of a new feature

Essential Monitoring Queries: Creating a Dashboard in ClickHouse Cloud

What's new in ClickHouse Cloud: spring 2025 roundup

How we built our Internal Data Warehouse at ClickHouse: A year later