Backpressure

Backpressure is the polite word for “we’re overwhelmed.” A producer is generating data faster than the consumer can process it. Something has to give. The queue between them either grows without bound (until memory runs out), drops messages (lossy), slows the producer (back-pressures), or fails (the 3am page). The pattern that wins, in stream processing as in plumbing, is to push the pressure back upstream rather than let it accumulate or vent.

Where the pressure comes from

Most pipelines aren’t built with explicit backpressure handling. They’re built assuming the consumer can keep up, and quietly fall apart when it can’t. The common causes:

• Bursty producers. A normal day looks fine; Black Friday looks catastrophic. Capacity sized for the median load drowns at the peak.
• Slow consumers. A downstream service takes longer per message than expected. The processor falls behind. Lag accumulates.
• External system slowdowns. Your processor calls an API; the API is slow today; your processor stalls; the upstream queue grows.
• Bad partition keys. One key generates 10x the volume of others. That partition’s consumer gets crushed. The rest are idle.
• Garbage collection pauses. The JVM stops the world for 30 seconds during a major GC. Lag spikes. Pressure builds.

The four strategies

When the consumer can’t keep up, there are four things you can do, each with trade-offs.

Buffer. Let the queue grow. Easy in the short term, fatal in the long. Memory runs out eventually; the broker disk fills eventually; the pipeline eventually fails harder than it would have failed earlier. Buffers are useful as shock absorbers for transient bursts, not as solutions for sustained overload.

Drop. Discard messages when the queue is full. Lossy by design. Acceptable for telemetry where the loss is statistically tolerable; unacceptable for transactional data. The trade-off has to be explicit, not accidental. The systems I’ve seen that quietly drop without anyone realising are the ones that lose customer trust when the discrepancy gets noticed.

Push back. Signal upstream to slow down. The producer eventually blocks (or buffers itself, which pushes the problem further up). This is “true” backpressure. It works only if the producer can be slowed down, which is true for most internal services and false for external sources you don’t control.

Scale. Add more consumer capacity. The default cloud-era answer. Works until you hit a partition limit (Kafka), a shard limit (Kinesis), or the downstream system’s ceiling (the database, the API, whatever’s constraining throughput).

The Kafka model

Kafka’s model is interesting because it doesn’t have backpressure in the classic sense. The broker doesn’t slow the producer; the producer just writes to the log at whatever rate it wants. Consumer lag accumulates if the consumer can’t keep up. The broker stores the messages until the retention period elapses, at which point old messages are deleted whether or not the consumer has caught up.

So “backpressure” in Kafka means “monitoring lag and scaling consumers,” not “telling producers to slow down.” The pressure goes into broker disk, then into retention loss. The architecture pushes the design responsibility to you: you have to monitor and scale, or accept that lag means loss eventually.

The reactive model

Reactive Streams, the spec that became the basis of RxJava, Project Reactor, and Akka Streams, takes a different approach. The consumer signals demand explicitly: “send me 10 more messages.” The producer can’t send faster than the consumer asks for. Backpressure is built into the protocol.

This works well within a single application but it doesn’t cross network boundaries easily. The TCP-level analog (sliding window, ACKs) does the same thing at the transport layer, which is why HTTP/2 and gRPC implicitly have backpressure even if you don’t think about it.

What actually works in production

The patterns I’ve seen survive contact with reality:

1. Bounded queues with explicit drop policies. When the queue fills, decide deliberately what to do. Drop the oldest? Drop the newest? Route to a dead-letter queue? The decision should be made at design time, not by the system silently.
2. Lag-based autoscaling. Consumer count scales with lag, not with CPU. Lag is the actual measure of “can we keep up.”
3. Dead-letter queues for the bad cases. When a message fails repeatedly, get it out of the main flow. Don’t let it block the consumer.
4. Rate limiting at the producer where possible. Especially important for retry storms after a downstream outage.
5. Circuit breakers for external dependencies. If the downstream API is failing, stop hammering it; switch to a degraded mode (cache, default, queue for later).
6. Adequate consumer parallelism. Kafka partition count should accommodate the consumer scale-out target, not the current load.

The thing that’s usually missing

Most teams haven’t run a load test that simulates a sustained burst. They’ve tested with normal-day traffic, which works. The first time the pipeline sees actual production peak, it tips over.

Game-day exercises are useful here. Simulate a 5x burst. Watch where the system breaks. Watch how it breaks – does it degrade gracefully or fail catastrophically? Most pipelines I’ve seen fail catastrophically the first time and degrade gracefully only after deliberate engineering. The engineering is unglamorous and gets deferred until the first 3am page makes it suddenly urgent.

Backpressure is one of those topics that’s deeply boring until it isn’t. Worth taking seriously before it becomes the conversation you’re forced to have at speed under pressure with someone who isn’t a fan of yours.

That’s February done. March moves to governance and contracts – what data contracts actually are, what they don’t fix, and why most lineage tools are quietly lying to you.