Data Contracts & Collaboration

In modern organisations, data moves across teams, tools, and systems faster than ever before. With this speed comes a persistent risk: the moment a producer changes the shape, meaning, or frequency of data without warning, downstream consumers can experience silent failures, broken pipelines, or bad analytics.

Data contracts are emerging as the solution, a formal agreement between data producers and data consumers that defines exactly how data will be structured, delivered, and governed.

What is a Data Contract?

A data contract is a specification that documents:

– Schema (structure of the data (columns, data types, constraints))

– Semantics (meaning of fields, business definitions, expected units)

– Quality Guarantees (thresholds for completeness, accuracy, freshness)

– Delivery Guarantees (how and when the data will be made available)

– Change Management (rules for making schema or semantic changes)


It’s like an API contract, but for data. And just like an API contract enforces communication standards between software components, a data contract enforces trust between teams exchanging datasets.

Why Data Contracts Matter

Without Data Contract

1. A data producer renames a column.
2. Downstream ETL fails, silently producing empty dashboards
3. Business decisions are made on incorrect data.

With Data Contract

1. Changes must be reviewed and approved.
2. Schema evolution is versioned and backward-compatible.
3. Data quality metrics are monitored and enforced automatically.

The Components of a Strong Data Contract

1. Schema Definition JSON Schema, Avro, Protobuf, or similar to define field names, types, constraints
2. Semantics Business meaning, domain definitions, units, enumerations
3. SLAs & SLOs Freshness (e.g., <5 min latency), completeness (e.g., >99%), accuracy thresholds
4. Ownership Who to contact for changes or incidents
5. Change Policy How breaking changes are communicated and rolled out
6. Versioning Maintaining old versions until consumers migrate

Implementing Data Contracts

1. Select a Format
   Many teams use machine-readable formats like Avro, Protobuf, or JSON Schema for the schema portion, with metadata stored in a governance tool such as Collibra (a too, I universally dislike, but that’s another story).
2. Integrate with CI/CD
   Treat data contracts like code — store them in Git, review changes via pull requests, and run automated checks.
3. Enforce at Ingestion
   Use validation tools to reject or quarantine non-compliant data before it enters the warehouse or lake.
4. Monitor and Alert
   Pair contracts with data observability platforms (e.g., Monte Carlo, Soda, Great Expectations) to track adherence to quality SLAs.

Benefits of Data Contracts

1. Reduced Breakages: Schema changes don’t catch consumers by surprise.
2. Improved Trust: Consumers know exactly what data means and how fresh it is.
3. Faster Iteration: Producers can evolve data safely with clear migration paths.
4. Regulatory Alignment: Easier to prove data lineage and quality for compliance frameworks (BCBS 239, GDPR, HIPAA).

Challenges of Data Contracts

Cultural Adoption — Requires producers and consumers to agree on responsibilities.

Overhead — May feel like “extra paperwork” unless automated.

Versioning Discipline — Must be treated as part of engineering, not an afterthought.

Example – Simple Data Contract

dataContractSpecification: 0.9.3
id: covid_cases
info:
  title: COVID-19 cases
  version: 0.0.1
  description: Johns Hopkins University Consolidated data on COVID-19 cases, sourced
    from Enigma
  links:
    blog: https://aws.amazon.com/blogs/big-data/a-public-data-lake-for-analysis-of-covid-19-data/
    data-explorer: https://dj2taa9i652rf.cloudfront.net/
    data: https://covid19-lake.s3.us-east-2.amazonaws.com/enigma-jhu/json/part-00000-adec1cd2-96df-4c6b-a5f2-780f092951ba-c000.json
servers:
  s3-json:
    type: s3
    format: json
    delimiter: new_line
    location: s3://covid19-lake/enigma-jhu/json/*.json
models:
  covid_cases:
    description: the number of confirmed covid cases reported for a specified region,
      with location and county/province/country information.
    type: table
    fields:
      fips:
        type: string
        required: false
        primary: false
        unique: false
        description: state and county two digits code
      admin2:
        type: string
        required: false
        primary: false
        unique: false
        description: county name
quality:
  type: SodaCL
  specification:
    checks for covid_cases:
    - freshness(last_update::datetime) < 5000d
    - row_count > 1000

dataContractSpecification: 0.9.3
id: covid_cases
info:
  title: COVID-19 cases
  version: 0.0.1
  description: Johns Hopkins University Consolidated data on COVID-19 cases, sourced
    from Enigma
  links:
    blog: https://aws.amazon.com/blogs/big-data/a-public-data-lake-for-analysis-of-covid-19-data/
    data-explorer: https://dj2taa9i652rf.cloudfront.net/
    data: https://covid19-lake.s3.us-east-2.amazonaws.com/enigma-jhu/json/part-00000-adec1cd2-96df-4c6b-a5f2-780f092951ba-c000.json
servers:
  s3-json:
    type: s3
    format: json
    delimiter: new_line
    location: s3://covid19-lake/enigma-jhu/json/*.json
models:
  covid_cases:
    description: the number of confirmed covid cases reported for a specified region,
      with location and county/province/country information.
    type: table
    fields:
      fips:
        type: string
        required: false
        primary: false
        unique: false
        description: state and county two digits code
      admin2:
        type: string
        required: false
        primary: false
        unique: false
        description: county name
quality:
  type: SodaCL
  specification:
    checks for covid_cases:
    - freshness(last_update::datetime) < 5000d
    - row_count > 1000

Data Contracts in the Data Mesh Era

In a data mesh architecture, data contracts are a critical enabler of the “data as a product” philosophy. Each domain team produces data products with clear interfaces (contracts) so that other domains can consume them without hidden dependencies or surprises. Data contracts are not about adding bureaucracy, they’re about enabling safe, predictable, and scalable data collaboration.

The analogy to software APIs is intentional: modern data platforms need versioned, testable, enforceable agreements between producers and consumers. Without them, the cost of change grows, pipelines break silently, and trust in data erodes.