Fintech Leader Juspay Analyzes Over 50 Million Daily Payment Transactions in Real-Time with ClickHouse

Juspay, an Indian fintech company, uses ClickHouse to power A/B testing and monitoring for its end-to-end payment solutions and real-time merchant dashboards. With over 50 million daily transactions for clients such as Amazon, Google, and Vodafone, ClickHouse was chosen for its ability to handle large volumes of data.

Juspay has established itself as a pioneer in the payments industry, consistently driving innovation in this space. With a mission to streamline online payments for merchants, Juspay acts as an intermediary between payment providers and merchant systems.

To ensure businesses have a seamless transaction environment, Juspay must provide monitoring and analytics services to guarantee that the payment system performs as expected. Juspay needed a solution that could help them make data-driven decisions.

With a diverse array of merchants and their ever-evolving needs, Juspay must keep up with frequent releases, often multiple releases daily, be it staggered or full-scale. Thus, they needed a monitoring solution that would enable them to maintain their rapid release pace while having the peace of mind that the latest release did not impact the running payment systems.

ClickHouse was selected as the solution to power Juspay's critical A/B testing and monitoring needs. As a payments orchestrator, Juspay has thousands of parameters related to payments, payment gateways, and different payment methods. Therefore, the potential impact of a new release on various dimensions is immense.

ClickHouse enables Juspay to compare the success rate percentage between version A and version B, and determine if a newer version is performing poorly by showing a lower success rate. In such cases, a rollback is conducted, which is especially helpful in instances where a new release is expected to improve performance but ends up affecting other services.

A traditional relational database was not capable of performing this level of analysis due to the massive volume of data generated by A/B tests. These tests are conducted across transactions from the last seven days, totaling around 600 gigabytes of data per day, equivalent to approximately 4TB. With ClickHouse, Juspay can group by all columns of low cardinality fields, which could amount to up to 10,000 dimensions with 50 million transactions a day.

A/B testing and monitoring with ClickHouse

For monitoring and A/B testing, Juspay uses a combination of Kafka, ClickHouse, and Grafana. They monitor active logs and perform transformations in Kafka, before utilizing Kafka queues to transfer data to ClickHouse via the Kafka engine.

Grafana is used to visualize and analyze A/B testing data

Juspay also needed a solution to provide their merchants with real-time analytics and a live view of their system performance. With ClickHouse, merchants can easily track their transactions and monitor their success or failure status in real-time.

The architecture of ClickHouse involves the use of a bash script CRON job for ETL (extract, transform, load) processes, which ingests data into the system. This data is then used to create custom dashboards for real-time analytics.

Real-time analytics dashboard for merchants created using ClickHouse

Juspay had previously been using BigQuery from GCP, but the cost of data scanning quickly added up, reaching over a thousand dollars per day. So, to reduce expenses, they sought a more cost-effective solution.

By switching this workload to ClickHouse, they achieved a 10x cost reduction in operating expenses. BigQuery is still used for certain types of queries, for example, when reporting on a particular id and where joins on ids are required. However, for all aggregate and real-time analysis they are using ClickHouse. Not only does ClickHouse solve their problems with impressive efficiency, it also requires significantly less infrastructure investment than BigQuery.

“ClickHouse solves most of our problems very efficiently at a small fraction of the price in terms of infrastructure. This is a far better advantage for us in our books,” said Tamil Selvan, Software Development Engineer on the project.

ClickHouse is a highly performant data platform, offering roll-ups and views that allow for controlled access to raw data. This helps prevent errors in data usage, such as joining high cardinality fields, and effectively manages resources.

“ClickHouse has many advantages; however we recommend not treating it as a black box. Although you can run any kind of query and get data, it's best to use ClickHouse in the proper way to make the most of all the advantages. If you follow the docs, you can't go wrong,” said Selvan.

By choosing ClickHouse, Juspay was able to reduce its operating costs and continue providing the best possible experience for its merchants. The advanced data processing capabilities and efficient cost structure of ClickHouse made it the clear choice.

Visit: https://juspay.in/