The word semantic is rather heavily used (often incorrectly) when discussing data models. The word semantic itself is an adjective relating to meaning in language or logic. When we think about its use in data models, we think of the…
Iceberg has gained a huge amount of popularity in recent years, but why is this table format now finding such widespread adoption? There are a number of reasons and I shall attempt to explain them in this post. Before Iceberg,…
Not going to win any awards for this blog post title. But keys really are a critical part of a database design and are often misunderstood, ignored and used in the wrong way. Keys are just attributes like everything other…
Is the answer AI? Ummm….not yet (correct at time of writing). Data engineering today looks remarkably different from five years ago. The role that emerged to build Hadoop clusters and write MapReduce jobs has evolved into something unrecognizable from its…
When WebAssembly launched in 2017, the pitch was clear – run native code in browsers at near-native speed. I sat next to an engineer at the time who was incredibly excited about this, but his explanation made no sense at…
Not a Wu-Tang Clan song. This is about Data Quality. Every organization claims to want high-quality data, but when pressed to define what that means, the conversation becomes vague. They was “clean data” or “accurate data” or “reliable data” –…
Memory is cheap, but it ain’t free. In the world of modern data engineering, compression is everywhere. It’s in your Parquet files, your Kafka messages, your database storage engine, your .txt file and your API responses. Yet despite its ubiquity,…
Writing this blog reminded me that I need a holiday. Anyway, there’s a class of data problem that shows up all over the place: On the surface, these seem like different problems. But they’re all the same problem in disguise:…
Honestly, we will do the conceptual and logical models later. Well, maybe. Probably. Yeah, we won’t. Data modelling is an art, but not everyone is an artist. We entust this highly specialised domain to engineers with (sometimes) little training and…
Every data platform eventually faces the same question: How do we actually get data from point A to point B? It’s a deceptively simple question with dozens of possible answers, each with its own trade-offs in terms of complexity, reliability,…
You must be logged in to post a comment.