Change Data Capture

Batch ETL has had a good run. Half a century of nightly jobs, scheduled extracts, full-table dumps, and morning surprises when something broke and the warehouse was empty at 9am. The pattern is so embedded in data engineering that it’s easy to forget it’s a workaround for the absence of something better. Change data capture is the something better, and it has quietly become the default in any data architecture built in the last few years.

What CDC actually does

Change data capture means reading the database’s transaction log directly, turning each committed change into an event, and emitting that event as a stream other systems can consume. Insert a row into orders; an event appears on a stream within milliseconds. Update a customer’s address; an event appears. Delete a record; an event appears. The downstream consumer sees the same logical changes the database saw, in the same order, without ever querying the source table.

This is fundamentally different from polling. A polling job reads the source table at intervals and compares to what it last saw. It misses changes that were inserted and updated between polls. It can’t see deletes unless you soft-delete. It puts load on the source. It runs at the cadence of the schedule, not the data.

CDC misses none of this because it’s reading the same log the database uses for replication. Postgres has its write-ahead log (WAL). MySQL has the binlog. SQL Server has the transaction log. Oracle has redo logs. Every serious relational database has had this for decades. CDC tooling just turns “you can replicate this” into “here’s a stream of every change.”

What Debezium got right

Debezium is the open-source project that made CDC accessible. It’s a set of Kafka Connect connectors that read transaction logs from Postgres, MySQL, MongoDB, SQL Server, Oracle, and a few others, and emit change events to Kafka topics. Each event has the row before the change, the row after the change, the operation type (c/u/d for create/update/delete), and metadata about when and where the change happened.

What makes Debezium good isn’t novel technology. It’s the right abstractions. The event format is standardised across databases. Schema changes propagate. Snapshotting is built in (a connector starts by reading the current state, then switches to following the log). It handles connector restarts gracefully. The pieces fit together.

The commercial alternatives – Fivetran, Airbyte, Striim, Confluent CDC connectors – all do versions of the same thing. The wire format differs, the operational story differs, but the core idea is identical: read the log, emit the changes.

What CDC unlocks

Once you have a stream of changes, several things become natural that were unnatural with batch.

Real-time analytics. Stream the changes into a warehouse or lake, and the analytics tables are seconds-behind, not hours-behind. The morning surprise becomes the morning sanity-check.

Replica databases without the database’s replication features. You can build a Postgres-to-Snowflake pipeline that’s logically a replica. Or a Postgres-to-Elasticsearch pipeline that’s a search index. Or Postgres-to-Redis as a cache. The source remains authoritative; the targets are views.

Event-sourced reads. The transactional database stays as the source of record. Downstream systems consume the event stream rather than querying the source. Read load on the OLTP database drops by an order of magnitude.

Audit logs you didn’t have to write. Every change is an event; the event stream is an audit log. Replay any point in history. Reconstruct any record. Answer compliance questions without having to instrument the application.

Microservice data sharing without the joints. Service A writes to its database. Service B consumes the change stream rather than calling Service A’s API. The coupling moves from synchronous API calls to asynchronous events. The architecture gets simpler.

The Iceberg connection

Why Iceberg

CDC into Iceberg is one of the more interesting architectures of the last two years. The change stream lands in an Iceberg table via an upsert pattern, the table maintains a current-state view via Iceberg’s merge-on-read semantics, and the historical change events are queryable as a time-series. You get warehouse-grade query performance on a table that’s within seconds of the source database, with full history retained. This was theoretically possible with Hive and Parquet a decade ago. In practice it was painful enough nobody did it.

Where CDC bites

It’s not free. The honest list of where it gets hard:

• Schema evolution. A column gets added on the source. The CDC connector needs to handle it. Downstream consumers need to handle it. The schema registry needs to know. Multiply by every schema change you ever ship.
• Initial snapshots. Bootstrapping a CDC pipeline from a billion-row table is non-trivial. Most tooling handles it, but the operational story is real.
• Backpressure. If the consumer falls behind the log, the database has to keep the log around. Postgres replication slots can hold WAL indefinitely, which eats disk and eventually breaks the database. You have to monitor this.
• Transactional boundaries. A transaction that updates three tables becomes three events on three topics. Restoring the transactional grouping downstream is your problem.
• Deletes. Soft deletes are easier. Hard deletes need explicit handling so downstream systems remove the record.

The pattern that wins

Batch ETL is becoming the special case. Most new architectures default to CDC for incremental data movement, with batch reserved for genuinely batch-shaped workloads (historical reloads, periodic aggregations, model training feature snapshots). The reasons are operational: CDC is more reliable, lower-latency, and lower-load than polling, and once you have the change stream you can do things polling could never do.

The legacy data warehouse architecture – nightly extracts, transform on landing, load into dimensional tables – is being replaced by CDC into a lake or lakehouse, with transformations running as the data arrives. dbt + CDC is a viable modern pattern. Iceberg + CDC + dbt is the same pattern with better storage. None of these need batch jobs except for the genuinely-batch-shaped work.

If you’re building anything new and your data engineering hasn’t looked at CDC, it’s worth a serious look. The pattern is mature enough that the operational story is well-understood, the open-source tooling is good, and the architectural simplification on the other side is substantial.

Next post – stream vs batch, why the lambda/kappa debate is over, and what replaced it.