Every piece of data in your system has a temperature, whether you’ve thought about it that way or not. Some data is accessed constantly, queried hundreds of times per second, requiring millisecond response times. Other data sits mostly untouched, accessed…
Data contracts are becoming the backbone of modern data architectures. As organisations shift from ad-hoc pipelines to product-oriented data ecosystems, they need guarantees: that data will arrive on time, in the right shape, and with the expected semantics. This is…
Databricks (like Snowflake) doesn’t rely on traditional B-trees, because it’s built on a cloud-native, columnar, distributed file architecture. It avoids B-trees entirely because the cost of maintaining per-row index structures would destroy the scalability benefits of its append-only, distributed Parquet…
The Kappa Architecture has emerged as a compelling alternative to the more complex Lambda Architecture. By treating all data as continuous streams and using a unified processing pipeline, it simplifies infrastructure and operations while delivering near real-time outcomes. Introduced by…
Bit of a departure from the usual data related posts but I think this one is important. Since its inception at Airbnb in 2014, Apache Airflow has become a cornerstone of modern data engineering. Formalized as an open-source project in…
While dimensional modeling has dominated data warehousing for decades, another approach has been quietly gaining traction in enterprises dealing with complex, rapidly changing data landscapes. Data Vault 2.0 represents a fundamentally different philosophy about how to organize and manage enterprise…
My personal experience with Data Fabric has been somewhat limited, as no organisation I’ve worked at has embraced this pattern. Speaking to my peers, I do see it being adopted at large enterprises, but it seems to lag behind the…
You must be logged in to post a comment.