In database systems, ACID – Atomicity, Consistency, Isolation and Durability is the cornerstone of transactional reliability. It guarantees that operations are safe, corruption-free, and recoverable. Yet, as data storage models evolve toward massive scale, distributed systems, and analytics workloads, new…
Streaming data systems and relational databases each offer immense value. Streaming excels at delivering low-latency, real-time insights; relational databases give strong consistency, rich transactional guarantees, and mature tooling. Yet, integrating these worlds without sacrificing usability or correctness has long been…
The term data product has become ubiquitous in modern data organizations, but its meaning often remains fuzzy. Teams talk about building data products, while creating the same old dashboards, reports, and datasets they’ve always built. Is this new Excel spreadsheet…
This is a notoriously challenging problem in the realm of data management. But why is it hard? It’s just dates and times. Data with timestamps is everywhere. Server logs record when events occurred. Sensors capture measurements at regular intervals. Financial…
Data contracts are becoming the backbone of modern data architectures. As organisations shift from ad-hoc pipelines to product-oriented data ecosystems, they need guarantees: that data will arrive on time, in the right shape, and with the expected semantics. This is…
The Kappa Architecture has emerged as a compelling alternative to the more complex Lambda Architecture. By treating all data as continuous streams and using a unified processing pipeline, it simplifies infrastructure and operations while delivering near real-time outcomes. Introduced by…
My personal experience with Data Fabric has been somewhat limited, as no organisation I’ve worked at has embraced this pattern. Speaking to my peers, I do see it being adopted at large enterprises, but it seems to lag behind the…
Test-Driven Development (TDD) has been a staple of software engineering for decades. It was first proposed by Kent Beck in the late 1990s and early 2000s. He popularized TDD as part of Extreme Programming (XP) and wrote the influential book…
In modern data architecture, clarifying the distinction between file formats (like Avro, Parquet) and table formats (like Iceberg, Delta Lake), as well as interoperability layers (XTable), is critical. Your choice guides performance, mutability, governance, and platform flexibility. This post gives…
When you run a modern database like Cassandra, RocksDB, or ScyllaDB, there’s a good chance it’s powered by a Log-Structured Merge Tree (LSM Tree). This data structure is built for one thing above all else: High-throughput writes at scale What…
You must be logged in to post a comment.