Exactly-Once Is a Lie (Mostly)

“Exactly-once” is the messaging guarantee that everyone wants and almost nobody actually has. Kafka claims it. Flink claims it. Spark Structured Streaming claims it. Most engineers I’ve met assume that ticking the “exactly-once” box in the config means the system will, in fact, process each message exactly once. It does not. It means something more specific and considerably less reassuring. Worth understanding properly before the first late-night incident.

The three guarantees

Messaging systems offer three flavours of delivery guarantee, classically:

• At-most-once. Each message is delivered zero or one times. No retries. If something goes wrong, the message is lost. Cheap, simple, useful for metrics-style data where loss is tolerable.
• At-least-once. Each message is delivered one or more times. Retries are used to guarantee delivery. The consumer might see duplicates. The default for most real systems.
• Exactly-once. Each message is delivered precisely once. No loss, no duplicates. Operationally what you wish you had. Theoretically achievable. Practically harder than the marketing implies.

Why exactly-once is theoretically hard

In any distributed system, you have producers, brokers, and consumers, and any of them can fail at any moment. The producer sends a message and the broker acknowledges it. The producer crashes before recording the ack. The producer recovers and resends. Now the broker has two copies. Or the broker sends a message to a consumer, the consumer processes it, and the consumer crashes before recording the offset. The consumer recovers and reprocesses. Same message, two side effects.

The Two Generals Problem applies: in a system where messages can be lost and parties can fail, there’s no way for both sides to definitively agree on whether a message was delivered without an unbounded sequence of acknowledgements. Exactly-once delivery, in the strictest sense, is impossible in an asynchronous distributed system.

So what does “exactly-once” mean when the vendors claim it? Two things, both worth understanding.

Exactly-once processing, not delivery

Kafka and most modern stream processors don’t actually deliver exactly-once. They deliver at-least-once, and use clever tricks to make the consumer’s observable behaviour look like exactly-once. The difference matters.

The standard mechanism is transactional offsets plus idempotent producers. Kafka producers have a unique producer ID and sequence numbers per partition. Broker-side de-duplication uses these to discard duplicate retries. On the consumer side, offsets are committed atomically with the output of processing, inside a Kafka transaction. If the consumer crashes mid-process, the transaction is aborted, no output is produced, and the original message is re-processed cleanly.

This works. It’s real exactly-once processing. But it requires that everything is inside the Kafka transaction, which means your side effects must be inside Kafka. The moment you write to an external system – a database, an HTTP API, an S3 bucket, anything outside the transaction – the guarantee evaporates. You have at-least-once processing of an external side effect, with all the duplicate-handling complexity that implies.

The idempotence escape hatch

The standard advice is “make your processing idempotent.” If processing the same message twice produces the same result, duplicates don’t matter. This is good advice as far as it goes. The problem is that it’s harder than it sounds.

Idempotence requires that every side effect is keyed by something stable in the message. Inserting a payment by message ID is idempotent. Inserting a payment with an auto-generated ID is not. Sending an email is idempotent only if you have an idempotency key the receiver respects. Calling an external API is idempotent only if the API supports idempotency keys (Stripe does, most don’t).

Most teams ship code that’s nominally idempotent and turns out not to be when the first duplicate hits in production. Race conditions in “check then insert” logic. Updates that compound (incrementing a counter is not idempotent, setting it to a value is). The corner cases multiply with the surface area of side effects.

What this means in practice

The honest framing is: design for at-least-once, achieve effectively-exactly-once through idempotence.

• Treat every message as potentially repeatable. Build the consumer accordingly.
• Use the message ID or content hash as a deduplication key for every side effect.
• Where idempotence is impossible (e.g., external APIs without idempotency keys), accept that you have at-most-once at the boundary, and design for the consequences.
• Use Kafka transactional offsets where you can keep the side effect inside Kafka (e.g., reading from one topic and producing to another). This is the one case where exactly-once-processing actually works.
• Stop ticking the “exactly-once” box and assuming the problem is solved.

The vendor incentive

If exactly-once is mostly impossible at the boundary, why do all the vendors claim it? Because it sells. “Exactly-once semantics” on a marketing slide reads like a guarantee. The fine print – that it’s only within the transactional boundary – is in the documentation that gets read after the first production incident. There’s a long history of distributed-systems vendors over-promising and the literature catching up later. CAP’s “consistency” was abused this way for a decade.

Confluent’s docs are actually pretty honest about this if you read them carefully. They distinguish exactly-once semantics (within Kafka transactions) from exactly-once delivery (to external systems, which they don’t claim). Most engineers don’t read that far.

The deeper lesson

Exactly-once is one of a small number of distributed-systems guarantees that sounds like it should work and doesn’t, in the same family as “strong consistency over a partitioned database” or “guaranteed delivery over an unreliable network.” The lesson isn’t to give up; it’s to know what the guarantee actually covers, and design for the cases where it doesn’t.

Distributed-systems engineers learn this the hard way, usually around 3am when a duplicate has done something irreversible. Better to learn it the easy way.

Next post – change data capture, the underrated pattern that’s quietly eating batch ETL.