Skip to main content
VectisFlow
VectisFlow
All insights

Data quality · 5 min

A contract that cannot stop a load is not a contract

A test that turns a tile amber and nothing else is documentation. A breach has to block a load or page an owner.

Platform engineering ·

Most data quality work ends at a dashboard. Tests run after the load, results are written to a table, a tile turns amber, and everybody agrees that someone should look at it. Nobody does, because looking at it is not anyone's job in particular.

A test with no consequence is documentation. It records an intention about the data and changes nothing about what reaches the warehouse, which means the wrong figure is served either way.

What the consequence actually buys

A contract is a claim about shape, range, freshness and referential integrity, declared beside the data and evaluated on every load. What makes it a contract rather than a report is the clause that fires on breach: the load stops, or a named person is paged.

That reads as severe until you price the alternative. A silent schema change propagates for weeks. Someone reconciles by hand in month three, and the cost is not the fix. It is every decision taken on the wrong figure in between, none of which can now be revisited.

Not every breach should stop a load

Blocking on everything is how a contract gets switched off, usually at two in the morning by whoever is on call. Grade the clauses instead, and be willing to defend the grading.

A null in a joining column blocks, because every downstream figure built on that load would be wrong. A freshness breach on a low-consequence feed pages the owner and lets the load through with the staleness recorded, because stale and labelled beats absent and unexplained.

  • Block: uniqueness, referential integrity, nulls in join keys, type changes
  • Page: freshness, volume anomalies, distribution shifts that need a person to judge
  • Record: everything else, kept so the pattern is already visible when it becomes an incident

The owner is the hard part

The engineering is a day of work. Naming the person the breach wakes takes considerably longer, because it is the moment a producing team discovers that a downstream consumer exists and has expectations of them.

That conversation is the point rather than an obstacle to it. A contract is an agreement between two teams that happens to be enforced in code, and one with a single signatory is a dashboard with better tooling.

Related

  • Data modelling · 6 min

    Slowly changing dimensions are a per-attribute decision

    Tracking history on everything is not caution. It is how a warehouse becomes something analysts route around.

    Platform engineering ·

    Read
  • Evaluation · 5 min

    Abstention is a feature

    A system that answers everything is a system nobody can rely on. The refusal is what makes the answers usable.

    Evaluation and assurance ·

    Read

Get in touch

Talk to us.

A first conversation runs about forty-five minutes and covers three things: what your estate actually looks like, whether anyone can define a correct answer or an agreed number, and whether your permission model resolves per user. Any one of them can rule the work out, and we would rather tell you in week one.

Response time
One working day, from a person who has read it.