Scaling Real-time Analytics to 1M Events

When we first built dataface.ai, we focused heavily on accuracy. Our SQL generation pipelines were solid, but as we onboarded our first enterprise customers, we hit a different kind of wall: scale.

Handling a few hundred queries per minute is easy. Handling one million events per second across distributed customer databases while maintaining real-time conversation latency is an entirely different beast. Here is how we redesigned our ingestion layer to survive.

The naive approach: synchronous polling

Initially, our platform would connect to customer databases and poll for schema changes or new metadata synchronously. This worked for a 10-table PostgreSQL database. It immediately collapsed when a customer connected a Snowflake warehouse with 15,000 tables.

Enter Kafka and Flink

We moved to an event-driven architecture using Apache Kafka. Instead of polling, we use Change Data Capture (CDC) via Debezium to stream schema changes and metadata updates directly into Kafka topics.

Apache Flink then consumes these streams, performs necessary transformations, and updates our metadata registry in Redis in real-time. This decoupling allowed our conversational agents to always have the latest schema context without ever issuing a blocking call to the customer's database.

The biggest win wasn't just performance—it was reliability. By decoupling ingestion from the conversational engine, an unexpected spike in schema updates no longer impacted user query latency.

Lessons Learned

• Always design for asynchronous metadata ingestion from day one.
• Partition your Kafka topics by customer ID to ensure fairness and prevent one massive warehouse from starving smaller clients.
• Monitor consumer lag religiously. We use Datadog to alert us the moment a Flink job falls behind.

Scaling is never truly finished, but moving to an event-driven ingestion pipeline gave us the headroom we needed to support our largest enterprise deployments.