There are some great names in software engineering and this is most definitely one of them. The lakehouse architecture promised to combine the best of data warehouses and data lakes.…
Iceberg has gained a huge amount of popularity in recent years, but why is this table format now finding such widespread adoption? There are a number of reasons and I…
Ever tried to arrange anything with friends via group chat? Messages arrive out of order, some people don’t respond, and others change their minds. Now swap friends with computers and…
Most data models answer a single question: what is true right now? That is fine until someone asks what was true last Tuesday, or what you believed was true last…
Not going to win any awards for this blog post title. But keys really are a critical part of a database design and are often misunderstood, ignored and used in…
Is the answer AI? Ummm….not yet (correct at time of writing). Data engineering today looks remarkably different from five years ago. The role that emerged to build Hadoop clusters and…
If you’ve ever queried a database with SQL (Structured Query Language) then you’ve been standing on the mathematical shoulders of Georg Cantor. His 19th-century work on set theory laid the…
When WebAssembly launched in 2017, the pitch was clear – run native code in browsers at near-native speed. I sat next to an engineer at the time who was incredibly…
Not a Wu-Tang Clan song. This is about Data Quality. Every organization claims to want high-quality data, but when pressed to define what that means, the conversation becomes vague. They…
Memory is cheap, but it ain’t free. In the world of modern data engineering, compression is everywhere. It’s in your Parquet files, your Kafka messages, your database storage engine, your…
Writing this blog reminded me that I need a holiday. Anyway, there’s a class of data problem that shows up all over the place: On the surface, these seem like…
Honestly, we will do the conceptual and logical models later. Well, maybe. Probably. Yeah, we won’t. Data modelling is an art, but not everyone is an artist. We entust this…
You must be logged in to post a comment.