Method
The mechanism, in the open.
We would rather be judged on the mechanism than on adjectives. This is the whole of it, the stages, the loops, the engagement shape and the first 90 days.
The pipeline
One spine, two branches, three loops.
The shared stages turn sources into modelled, tested data. From there the work forks, one branch to analytics and the semantic layer, one to retrieval and generation. Both rejoin at assurance, because both have to be checkable.
01 · Survey
Open the estate before designing anything
A stratified sample of what is actually there. On the warehouse side that is table volumes, key integrity, duplication and the reconciliation gaps nobody has written down. On the document side it is formats, scan quality and revision patterns. Both sides carry a permission model each source enforces, and two or three subject areas or document families usually account for most of the value.
Hard partSkipping this is the most expensive decision available. Almost every stalled programme we are asked to look at was designed against an assumed estate rather than a sampled one.
02 · Ingest
Incremental, idempotent, resumable
Connectors with change-data capture where the source supports it and content hashing where it does not. Reprocessing a document after a parser change must produce one record, not two.
Hard partIdempotency is easy to claim and hard to hold once three sources disagree about what a document's identity is.
03 · Parse
Recover structure, in feeds and in documents
Extracts are decoded against a declared schema, with types, encodings and null conventions pinned rather than inferred fresh on every load. Documents get native text where a layer exists, OCR where it does not, and layout-aware parsing for anything tabular. Tables and appendices carry a disproportionate share of the answers and are what naive extraction destroys first.
Hard partRouting by source quality. Sending everything to the expensive parser is wasteful; sending everything to the cheap one silently loses the hardest third of the estate.
04 · Model
Decide what a customer is before counting them
Dimensional models where the questions are known, wide tables where they are not, and entity resolution across the systems that each hold part of the answer. Slowly changing dimensions are decided per attribute rather than as a blanket policy.
Hard partTracking history on every attribute is the default that quietly makes a warehouse unqueryable. Deciding which attributes actually need it is unglamorous, and it is the difference between a model people use and one they route around.
05 · Quality
Contracts that stop a load, not a dashboard
Shape, range, freshness and referential expectations declared beside the data and run on every load. A breach blocks the load or pages an owner, because a failing test nobody is accountable for is documentation rather than control.
Hard partThe tempting fix when a source changes shape is to coerce it back to the old one. That keeps dashboards green while the numbers stop meaning what they meant, and it is usually found months later by someone reconciling by hand.
06 · Semantic
One definition of the metric, with an owner
Metric and dimension definitions in one place, versioned, consumed by every tool rather than reimplemented in each. The layer is the contract between the model and the people who ask it questions.
Hard partEvery organisation believes it has one definition of revenue until the definitions are written down next to each other. Reconciling them is a political exercise before it is a technical one, and pretending otherwise is how the layer stalls.
07 · Publish
Marts and reports people can be held to
Consumer-facing datasets with a freshness SLA, a named owner and lineage back to source. A published figure carries the same obligation as a generated answer: you must be able to say where it came from and when it was last true.
Hard partPublishing is where scope grows without anyone deciding to grow it. Every new consumer adds a dataset that must be kept correct forever, so the entry test is whether someone will act on it, not whether someone asked for it.
08 · Chunk
On structure, not on token count
Sections, clauses and conversational turns, with parent context attached and overlap tuned against a retrieval evaluation set. Fixed windows are a default, not a decision.
Hard partChunk boundaries are the most under-examined choice in most retrieval systems. Severing a finding from the identifier that gives it meaning costs more recall than any embedding-model upgrade will win back.
09 · Enrich
Typed metadata so retrieval can filter before it ranks
Entity resolution across sources, date and revision normalisation, document-type classification. A hybrid filter on type and date routinely beats a better embedding model, at a fraction of the cost.
Hard partEntity resolution across systems that each have their own idea of an identifier. This is ordinary data engineering and it is where most of the unglamorous effort goes.
10 · Index
Vector and lexical, with permissions inside
Dense embeddings alongside a lexical index, with source ACLs stored per chunk. Rebuilds run blue-green so a re-embedding never takes retrieval offline.
Hard partPermissions belong in the index, not in a post-filter. A post-filter leaks the existence of documents a user cannot open.
11 · Retrieve
Hybrid, pre-filtered, reranked
Metadata pre-filtering, hybrid dense and lexical candidate generation, cross-encoder reranking. Tuned against a labelled set built from the questions people actually ask.
Hard partResolving access per user at query time, at latency. If your directory cannot answer group membership fast enough, this is where you find out.
12 · Ground
Cite the passage or abstain
Every claim carries a citation to a specific chunk of a specific revision. Claims without a supporting passage are refused rather than improvised.
Hard partAbstention has to be designed for, and defended. It is the behaviour that earns trust with a technical buyer and the first thing a demo-driven roadmap removes.
13 · Evaluate
A number, per document family, in CI
A labelled question set, scored per family, run as a build gate. Retrieval and generation measured separately, because they fail separately.
Hard partAggregate scores hide the family that fails completely. Stratify or you will ship a system that works well on average and not at all for one team.
14 · Review
Humans where the error is expensive
A review queue above a consequence threshold, a feedback control on every answer, and a sampled audit. Every adjudication writes back to the labelled set.
Hard partReview effort should fall over time. If it does not, the pipeline is not learning, and you want to know that in month two.
Reference architecture
What gets built, in five layers.
Sized to your estate rather than to a template, and deployed inside your boundary. The assurance layer is the half most projects under-resource and the half that decides whether the system can be switched on.
01 · Sources
- Document management
- Case & ticket systems
- Shared drives
- Transcripts & email
Read-only connectors. ACLs read at the same time as content.
02 · Pipeline
- Parse & OCR routing
- Structure-aware chunking
- Entity & revision resolution
- Embedding
Idempotent, resumable, versioned. Every record traces to a source revision.
03 · Indexes
- Vector index
- Lexical index
- Metadata & ACL store
- Lineage store
Blue-green rebuilds. Permissions live here, not in a post-filter.
04 · Serving
- Pre-filter by ACL & metadata
- Hybrid retrieval
- Rerank
- Grounded generation
Access resolved per query against your directory, before ranking.
05 · Assurance
- Labelled question set
- CI regression gates
- Adversarial & permission suites
- Live sampling
The half most projects under-resource, and the half that decides deployability.
Engagement shape
How engagements are shaped
We are engaged to build and hand over, not to stay. Every engagement is scoped so that the people who will run the system afterwards have changed it themselves before we leave.
01
Discovery, 2 to 3 weeks
A survey of the real sources, structured and documentary alike, with permission modelling and an evaluability screen. Ends with a written recommendation, including the recommendation not to proceed where that is the honest answer.
02
Build, 8 to 16 weeks
Two-week increments against a working system, not a sequence of documents. Your engineers are in the repository from the first increment; a handover that begins at the end is a handover that fails.
03
Assurance, in parallel
The harness is built alongside the pipeline, not after it, whether it measures answers or the numbers behind them. Nothing goes to users without a measured baseline it can be compared against.
04
Handover and support
Runbooks, a rollback path that has been executed at least once against real data, and a defined support window that ends. If you need us permanently, something has gone wrong with the handover.
First 90 days
What the first 90 days looks like
Concrete enough to hold us to. Dates shift with the size of the estate and with access lead times; the sequence does not.
Days 1 to 15
Survey and access
- Stratified sample of the real sources, by format, era and owning system
- Permission model mapped, including whether it resolves per user at query time
- Access, environments and security review under way, usually the long pole
- Evaluability check: what does correct look like, and who adjudicates it
Days 16 to 30
Baseline and first slice
- A labelled set of 200 to 400 items, built with the people who ask the questions
- Ingestion for the highest-value source only, one subject area or one document family
- A measured baseline, deliberately unimpressive, and the number every later change is compared against
Days 31 to 60
Build against the number
- The branch the engagement needs, tuned against the labelled set: semantic layer and marts, or chunking, enrichment and retrieval
- Permission inheritance implemented and covered by a synthetic-user regression suite
- What the system refuses to do agreed with the people relying on it: a load that blocks, an answer that abstains
- Evaluation running in CI with thresholds that fail a build
Days 61 to 90
Production and handover
- Deployment into your environment, behind your identity provider, inside your boundary
- Review queue and feedback loop live, writing corrections back into the labelled set
- Monitoring on freshness and quality per dataset, and on abstention rate and cost per answered question
- Runbooks, rollback executed once against real data, and your team making changes unaided
Principles
Five things we will not trade away.
Mechanism over adjectives
Every claim we make about a system should be checkable by reading the pipeline. If it cannot be checked, it is marketing and we will leave it out.
Measure before you improve
No change to a pipeline before there is a baseline to compare it against, whether the output is a published number or a grounded answer. Improvements that cannot be demonstrated are indistinguishable from preferences.
Abstention is a feature
A system that says it does not know is deployable in places a system that always answers is not.
Build on what you already run
The fastest way to make a system unsupportable is to introduce a stack your team has no on-call experience with.
Leave with the lights on
Documentation, runbooks and a rollback path that has actually been executed. The measure of a good handover is that nobody needs to call us.
Start here
Bring us an estate and a constraint.
The first conversation covers three things: what the estate actually looks like, whether anyone can define a correct answer or an agreed number, and whether the permission model resolves per user. Any one of them can rule the work out, and we would rather find that in week one.
- Prefer email
- hello@vectisflow.com
- Response time
- One working day, from a person who has read it.