Kafka consumer lag is not a metric — it is a symptom

The setup

A platform processing 1.2 million events a day. Consumer lag on the single busiest consumer group had climbed to 4.2 million messages and kept climbing. The instinctive move is to add brokers — which costs money, adds rebalancing pain, and does not fix anything.

Lag is a queue. Queues only grow when arrival rate exceeds processing rate, or when work is distributed unevenly. Before touching hardware, we asked which of the two it was.

It was the keys

The producer was partitioning on customerId. Ninety-one of our top two hundred customers accounted for 63% of all traffic, so sixty partitions were idle while three carried everything. Every event for one customer landed on one partition and had to be processed strictly in order.

We changed the partition key to a composite — customerId:eventType — which spread load without breaking per-type ordering guarantees. No broker count change. Lag dropped to under 3,000 within the hour and stayed there.

The three checks, in order

• Is the load even? Per-partition lag tells you immediately. One fat partition is a key problem; all partitions high is a throughput problem.
• Is a single consumer hot? If one instance in the group is 10x behind its siblings, the bug is in that consumer, not the cluster.
• Are you processing in batch? Default consumer fetch settings often starve you. Tuning max.poll.records and fetch.min.bytes is free performance.

What we changed in the pipeline

1. Composite partition keys        -> lag 4.2M -> 3K
2. max.poll.records 500 -> 2000   -> throughput +180%
3. Dead-letter topic per group    -> silent failures -> zero

The lesson generalises beyond Kafka: when a queue grows, measure its distribution before you buy more capacity. Almost every time the answer is in how the work is partitioned, not how much machine is underneath it.