In today’s era of big data, organizations face the challenge of processing vast datasets with varying timeliness and accuracy requirements. The Lambda Architecture, introduced by Nathan Marz, offers a solution by combining batch processing (for accuracy and reliability) with stream…
Change Data Capture (CDC) is one of those concepts that seems deceptively simple: Just capture the changes But it soon becomes complex once you start implementing at scale. In today’s world of streaming analytics, real-time machine learning, and event-driven microservices,…
For over a decade (I had to check this as it made me feel old) Pandas has been the go-to Python library for data analysis. Its DataFrame API has shaped how millions of analysts, scientists, and engineers work with tabular…
In database systems, ACID – Atomicity, Consistency, Isolation and Durability is the cornerstone of transactional reliability. It guarantees that operations are safe, corruption-free, and recoverable. Yet, as data storage models evolve toward massive scale, distributed systems, and analytics workloads, new…
The last five years have seen a number of open data table formats vying for position, these include Apache Iceberg, Delta Lake and Apache Hudi. By mid-2025, the winner is clear: Apache Iceberg. This is not just a technical victory…
In other words, Data Mesh. Yes, another technology buzz word to add to the collection? But this one is different. The biggest challenge in building a data mesh is not technology. It is communication, culture, and trust. Convincing colleagues meant…
We all understand the concept of a marketplace. But it’s usually for things we spend money on to acquire. In today’s digital economy, data also has value. Real, tangible value realised through data products, analytics and whatever AI is capable…
I have previously had exposure to Domain Specific Language (DSL) using Xtext (). The DSL was used to define context and models. It meant we were able to give the business users and SMEs the tools they needed to build…
This is a notoriously challenging problem in the realm of data management. But why is it hard? It’s just dates and times. Data with timestamps is everywhere. Server logs record when events occurred. Sensors capture measurements at regular intervals. Financial…
Data is everywhere, but meaning is scarce. Businesses increasingly struggle with silos, databases older than most people operating them (hello DB2), inconsistent or missing semantics, and limited discoverability. Knowledge Graphs provide a unifying framework, linking data entities and relationships in…
You must be logged in to post a comment.