How Trigger.dev is using ClickHouse to scale observability for long-running AI workflows

A user uploads a video and expects a transcript minutes later. An AI agent gathers context across dozens of systems before responding. A workflow fans out into hundreds of subtasks, reading documents, calling APIs, and transforming data long after a request leaves the browser.

In a world driven by AI workflows, what once looked like simple background jobs have become complex, long-running processes that power the core behavior of modern applications.

This is where Trigger.dev comes in. As co-founder and CTO Eric Allam puts it, “We’re your one-stop shop for deploying serverless asynchronous tasks. You write code and deploy it to our infrastructure, and we orchestrate it, execute it, retry it, queue it, and make sure it runs reliably.”

Founded in London in 2022 as an open-source project, Trigger.dev initially focused on helping developers run background automation more easily. But as AI workflows and agents moved into production, the platform’s role expanded. Customers increasingly relied on Trigger.dev not just for automation, but as a core layer inside their applications, coordinating long-running computation, large-scale data processing, and complex execution chains.

As adoption grew, so did the volume of telemetry generated by those workflows. Every task execution produced logs, traces, and events across thousands of tenants, turning observability itself into one of the platform’s biggest technical challenges.

We caught up with Eric to learn about Trigger.dev’s evolution alongside the rise of AI workflows, what led the team to rethink their database architecture, and why they chose ClickHouse.

Outgrowing Postgres #

Trigger.dev’s initial setup took an approach familiar to many open-source projects: “Let’s just use Postgres and see how far we go.”

Early on, it worked well. Users could spin up Trigger.dev with a simple Postgres instance and immediately begin running tasks while the platform collected logs and telemetry behind the scenes. But as adoption grew, the volume and nature of that data began to change.

“For some of our bigger customers, and especially our cloud customers, the Postgres implementation wasn’t really scaling,” Eric says.

Every task execution generated logs, traces, and events, all flowing into a single table that, as Eric puts it, “kept growing and growing and growing.” The team introduced partitioning and other optimizations to keep things manageable, but they kept running into limits. Adding indexes became increasingly impractical, queries slowed, and even basic questions like counting rows or understanding usage patterns became hard to answer reliably.

At the same time, the database was becoming expensive to operate and risky to modify. Heavy analytical workloads could threaten the operational stability of the system responsible for running customer tasks, creating tension between reliability and visibility.

Another challenge was access patterns. As Eric explains, Trigger.dev needs to analyze “extremely dynamic” observability data across thousands of tenants, each generating different types of events and attributes. Supporting new analytics features in Postgres would have meant building separate pipelines and aggregation systems, adding even more complexity to an already strained architecture.

With telemetry volumes continuing to climb, it became clear that Postgres, while well suited for operational workloads, wasn’t the right foundation for large-scale observability analytics.

Why they chose ClickHouse #

The team kicked off their search for a new database, prioritizing performance, scalability, open-source compatibility, and flexibility across deployment models. As Eric puts it, “Anything we chose would have to work for our self-hosters and our cloud customers.”

At first, they explored staying within the Postgres ecosystem. Emerging OLAP extensions and Postgres-adjacent options like DuckDB and ParadeDB promised analytical capabilities without abandoning familiar tooling. But the team worried that analytics would remain too tightly coupled to their operational database. “Anything non-operational that could potentially take down the operational side of the database is risky,” Eric says. Running a separate analytical Postgres instance would add complexity without really solving the underlying problem.

Around the same time, they began hearing from peers at developer tooling companies like Baseline, Highlight, and Langfuse, who were using ClickHouse to handle similar observability workloads at scale. Those conversations reinforced what they were seeing across the broader open-source ecosystem: a growing number of observability-focused projects successfully running ClickHouse in production. Being able to study real-world schemas and access patterns gave Eric and the team confidence that the database matched their workload.

Ultimately, ClickHouse offered everything Trigger.dev needed: proven performance at scale, a familiar SQL interface, a rapidly growing ecosystem around observability workloads, and an open-source model aligned with their self-hosted and cloud deployments. “We decided ClickHouse is the way to go,” Eric says.

A seamless migration #

Even after deciding to move to ClickHouse, the team didn’t migrate immediately. As Eric explains, they were watching the database evolve, waiting for improvements like ClickHouse’s native JSON data type, which would better support the highly dynamic telemetry generated by Trigger.dev’s workloads. “We saw it was close on the horizon,” he says.

Once they felt confident the database matched their needs, the team began migrating observability workloads using a gradual approach designed to avoid disruption.

The base unit of Trigger.dev’s platform—a single execution of a task—is called a “run.” During the transition, logs were routed at that level, allowing each run’s data to be written either to Postgres or ClickHouse. This gave the team a way to validate performance and correctness under real production conditions while keeping the platform fully operational.

New workloads increasingly flowed into ClickHouse, while a shrinking portion of legacy runs continued writing to Postgres as the migration progressed. The platform stayed fully operational throughout, requiring no downtime or customer intervention. “It was seamless to the user,” Eric says. “They didn’t have to do anything—it all happened in the background.”

Real-time insights at scale #

With observability workloads now running on ClickHouse, Trigger.dev has a much clearer view into how the platform is being used. Queries that were impractical or too risky to run on Postgres have become routine, giving the team direct insight into real-world usage.

As Eric explains, the team can explore telemetry, analyze usage patterns, and build internal insights without worrying about impacting operational systems. “Before, we didn’t even know how many rows there were,” he says. With ClickHouse, they can understand how logs flow through the platform in real time.

That visibility has translated into product improvements. Using ClickHouse’s materialized views, Trigger.dev introduced aggregated usage metrics that show customers how many logs they generate and how activity changes over time. “This was extremely simple and took maybe an hour in ClickHouse,” Eric says. “To do that in Postgres would have been a huge pain.”

The team has also seen a major reduction in storage footprint. “Just looking at how big datasets are in ClickHouse versus before, it’s a massive difference,” Eric says.

Performance, too, has remained strong as data volumes continue to grow. With ClickHouse, queries retrieving tens of thousands of rows return with a P95 latency of around 200 milliseconds. “We’re pretty happy with that,” Eric says.

Building without limits #

In the long run, moving observability to ClickHouse was as much about fixing bottlenecks as it was about removing constraints on what they can build into the product.

In Postgres, practical limits shaped the dashboard experience. Large volumes of telemetry had to be constrained, and exploring logs across complex workflows wasn’t always feasible. With ClickHouse, those limits are starting to disappear. As Eric explains, the goal is to reach a point where developers can explore runs freely, without artificial caps. “What we want to do isn’t really possible without that,” he says. “We’re working toward a dashboard experience where there are basically no limits on how many logs you can view at once.”

That aligns with Trigger.dev’s broader vision. As outlined in the company’s recent roadmap, upcoming improvements—from expanded logging and log streaming to advanced metrics and new agent tooling—aim to make production AI workflows easier to build, debug, and scale. Observability becomes part of the developer experience, not just supporting infrastructure.

In Eric’s words, ClickHouse strikes a balance between “fast, scalable, and familiar,” allowing the team to evolve the platform without forcing users to rethink how they work with data.

As AI workflows grow longer-running and more complex, that foundation matters more than ever. The more applications rely on autonomous agents and distributed execution, the more visibility developers need into what those systems are actually doing. With ClickHouse powering observability, Trigger.dev is building toward a future where workflows can keep growing in complexity and scale without sacrificing visibility or control.

Looking to scale observability without limits? Try ClickHouse Cloud free for 30 days.