Making a data platform trustworthy enough to publish from

Context

A government department publishes an official statistic on a fixed calendar, assembled from operational systems that other parts of the department own and change.

When a published figure turns out to be wrong, the correction is itself published, with a notice explaining what changed. That is the standard the platform has to meet, and it is a higher one than not falling over.

The problem

A quarterly release was corrected six weeks after publication. A status code in an upstream case system that had recorded a case as closed had been redefined to record it as closed or transferred, and the transferred volume was large enough to move the headline figure.

Nothing failed. The column was still a string of the same length, still inside the allowed values written down years earlier, so the load accepted it. The pipeline had no way to notice that a value had changed meaning while keeping its shape.

Finding the cause then took nineteen days of people reading SQL by hand. Lineage existed as documentation written during an earlier programme rather than as something the platform emitted, so it described a pipeline that had since moved on.

Why earlier approaches failed

A data quality dashboard was already running. It tracked row counts, completeness and null rates, all of which looked entirely normal throughout, because the defect changed what values meant without changing how many of them arrived.

A data catalogue had been populated by hand under the same earlier programme. Its entries were accurate on the day each was written and drifted from there: on a sample taken in week one, 41% of column descriptions no longer matched the column they described.

The pipeline we built

Quality checking moved from a dashboard beside the pipeline to a gate inside it. A check that cannot stop a load is a report, not a control.

1. 01Source agreement

   A written contract per feed, agreed with the team that owns the source system: fields, types, allowed values, the meaning of every coded value, and a named owner who tells us before any of it changes. Six of the eleven feeds had no identifiable owner when we started, and finding them was slower than writing the contracts.

2. 02Contract enforcement

   Checks run against landed data before anything is loaded: schema, type, allowed value set and referential integrity. A breach quarantines the batch and raises an incident against the named owner, so a change upstream becomes a conversation rather than a silent coercion.

3. 03Distribution monitoring

   The check that would have caught the original defect. The share of each coded value is compared against a trailing baseline, with a tolerance agreed per field with the statisticians who use it. Replayed against the archived batches, the redefined status field breaches on the first load after the change.

4. 04Lineage capture

   Column level lineage emitted by the transformation layer rather than maintained by anyone. Every published figure resolves to the columns, transformations and source batches that produced it, and the resolution is a query rather than an investigation.

5. 05Freshness SLAs

   Each feed carries a stated freshness and each publication states the freshness it requires. A feed that misses its SLA marks every figure downstream of it as stale before anyone starts assembling the release, which moves the discovery from the day of publication to the day of the miss.

6. 06Reconciliation to source

   Counts and control totals are compared back to the operational system rather than to the previous run of the pipeline. Comparing a copy against an earlier copy of itself reconciles cleanly through exactly the class of upstream error that caused the correction.

7. 07Publication gate

   A release cannot be produced until every contributing feed is inside its freshness SLA, every contract check has passed and every quarantined batch has been resolved by its owner. The gate is a build step and it fails the build, so the decision to publish anyway is explicit and recorded.

8. 08Incident drill

   Once a quarter a feed is deliberately altered in a copy of the platform and the team is timed on identifying which upstream change moved which figure. It is the only measurement that shows whether the lineage works when someone actually needs it.

What shipped

• Enforced contracts on all eleven source feeds, each with a named owner in the team that owns the source system and an agreed change notification path.
• A publication gate in the release pipeline that fails the build on a contract breach, a missed freshness SLA or an unresolved quarantine.
• A column level lineage store, queryable from any published figure back to the source batches behind it.
• A quarterly incident drill with a timed target, owned and run by the department's data engineering team.

Outcomes

Each figure below carries the method behind it and the baseline it is measured against, which is the form a result has to take before it means anything.

Time to identify the upstream cause of a moved figure

Median across the first four quarterly drills, each run against a deliberately altered feed in a copy of the platform. The baseline is the nineteen days the same question took by hand during the correction that prompted the engagement.

19 days → 31 min

Loads blocked by a contract breach

Across the first two publication cycles after the gate went in. Four were allowed value breaches, two were distributional shifts in coded values and one was a type change. Two of the seven would have altered a published figure had they loaded.

7 blocked, 2 figure-affecting

Published figures traceable back to source rows

All 94 figures in the two most recent publications, traced from the published cell to the source rows and the transformation version that produced them. The baseline is the same trace attempted by hand during the correction described here, which reached source for 11 of the 94 before the team ran out of time.

11 of 94 → 94 of 94

Column descriptions matching the column

All 214 columns across the eleven feeds, checked against the data they actually carry. The baseline is the hand maintained catalogue sampled in week one; descriptions now come from the contracts and are verified on every load, so drift fails a check rather than accumulating.

59% → 214 of 214

What next

The same contract and lineage stages are being applied to two further publications the department produces, reusing the checks and the gate and replacing only the source agreements.