Ingesting clickstream data from 45M users into ClickHouse

Jordan Thoms, CTO and Co-Founder at Kami

Kami's in-house analytics system collects clickstream data for 45 million users, reaching 300 GB per day and over 200 billion total data points. This data, ingested into the company’s data warehouse, informs key business decisions, such as product roadmaps, sales planning, and marketing. With Kami’s explosive growth, we have successively rearchitected our data infrastructure to allow for massive scale. In this talk, Jordan will explain how user activity data from Kami’s web app moves through Ruby on Rails, Sidekiq, Kafka, and an in-house Kafka consumer before real-time ingestion into ClickHouse.