Assembling regulatory evidence from casework correspondence in central government

Context

A central government agency conducts investigations whose evidence base is correspondence: submissions, responses, internal assessments and file notes, accumulated over eleven years across two case management systems and a shared drive that predates both.

Case officers assembling an evidence bundle were reading across all three by hand. The work was slow, and its consistency depended on which officer did it.

The problem

The agency's obligation is not to find documents but to assemble a defensible position: every assertion in a bundle supported by a citation, with the provenance of that citation intact if it is challenged years later.

Volume made that expensive. A mature case could touch several thousand documents, of which perhaps forty mattered, and identifying those forty was expert work that could not be delegated.

The constraint that shaped everything: nothing could leave the agency's boundary, and every automated step had to be explainable to an internal auditor in terms of what it did and what it might have missed.

Why earlier approaches failed

A keyword-and-taxonomy approach had been in place for years. It required officers to know the vocabulary a submission used, which in practice meant the same substantive point was missed whenever a correspondent phrased it unusually, precisely the correspondence most worth finding.

A commercial e-discovery tool was evaluated and rejected on two grounds: it required data egress the agency could not permit, and its relevance scoring could not be explained in terms the agency's auditors would accept.

The pipeline we built

Built entirely inside the agency's boundary, with explainability treated as a delivery requirement rather than a documentation task.

1. 01Ingestion

   Incremental connectors to both case management systems and the shared drive, idempotent by content hash so a document reprocessed after a parser change produces one record rather than a duplicate.

2. 02Threading

   Correspondence reassembled into threads across systems using headers, quoted-reply detection and participant resolution. A single message is frequently meaningless without the exchange it sits in, and thread reconstruction changed retrieval quality more than any model choice made later.

3. 03Chunking

   On turn boundaries within a thread, with the thread summary and participant roles attached to each chunk. Sub-turn splitting only for the small number of very long submissions, on paragraph boundaries.

4. 04Typed extraction

   A constrained extraction stage producing typed records: assertion, date, party, obligation cited, each carrying the exact source span. Extraction is schema-validated; anything that fails validation goes to a review queue rather than being silently dropped.

5. 05Retrieval

   Hybrid retrieval with metadata pre-filtering on case, date range and party. Case boundaries are enforced at the index level, so a query against one case cannot reach another's material even when the phrasing is identical.

6. 06Evidence assembly

   Candidate passages assembled into a draft bundle grouped by assertion, each with its citation and a confidence band. A case officer accepts, rejects or annotates every item; nothing enters a bundle unreviewed.

7. 07Lineage

   Every extracted record carries its source document, revision, pipeline version and model version. Reproducing why the system surfaced a passage two years ago is a lookup rather than an investigation.

8. 08Evaluation

   420 labelled questions built from closed cases where the correct evidence set was already known, scored on recall of known-relevant passages: the measure that matters when the cost of a miss is asymmetric.

What shipped

• An extraction and assembly pipeline running wholly within the agency's boundary, with model inference through a private endpoint.
• An evidence review interface with accept, reject and annotate, feeding every decision back into the labelled set.
• A lineage store making any surfaced passage reproducible to a pipeline and model version.
• An assurance pack for internal audit: method, measured recall by document family, known failure modes and mitigations.

Outcomes

Each figure below carries the method behind it and the baseline it is measured against, which is the form a result has to take before it means anything.

Officer time per evidence bundle

Median hours from case selection to a reviewed bundle, across 40 mature cases before and 40 after.

31 h → 9 h

Recall of known-relevant passages

Measured on closed cases where the correct evidence set had already been established by hand. The baseline is the keyword-and-taxonomy approach it replaced.

64% → 91.4%

Extraction records amended at review

Share of typed records an officer corrected, tracked monthly as reviewer corrections fed back into the labelled set.

18.3% → 6.1% over six months

Internal audit findings on method

At the first internal audit following go-live. Two advisory recommendations were made, both on retention of intermediate extraction records.

0 findings, 2 recommendations

What next

The agency is extending the same pipeline to a second casework directorate, reusing the threading and lineage stages unchanged and replacing only the extraction schema.