Service 01
Data and platform engineering
Most AI programmes stall on data nobody fixed. So do most analytics programmes. We build the platform both of them depend on, and we build it to be handed over.
The problem this solves
The warehouse was built for a set of questions somebody had five years ago. Since then three source systems changed shape, two teams built their own extracts because the central one was too slow, and the number in the board pack no longer matches the number in the operational report. Nobody can say which is right without an afternoon of manual reconciliation.
The usual response is a migration, and migrations usually ask for a freeze. The business will not grant one, so the work runs at half attention alongside the platform it is meant to replace, and the two drift.
Underneath that sits a quieter problem. Almost nothing in the estate declares what it expects. A source changes a column type, the load coerces it, and dashboards stay green while the figures stop meaning what they meant. That is usually found months later by someone reconciling by hand.
Method
How we build it
Every stage below can be evaluated on its own, which is what makes a platform handoverable. If you cannot tell whether a number moved because the source changed, the model changed, or the load silently coerced something, you do not have a platform, you have a habit.
01 · Estate survey
A sampled inventory before any build: row counts against what the business believes, update patterns, the joins that actually get used, and which of the three systems holding a customer record is treated as correct when they disagree.
02 · Ingestion and orchestration
Change-data capture where the source supports it, content hashing where it does not. Backfill and incremental run through the same code path, because two paths drift and only one of them is ever tested.
03 · Modelling
Dimensional models where the questions are known, wide tables where they are not. Slowly changing dimensions are decided per attribute rather than as a policy, because tracking history on everything is how a warehouse becomes unqueryable.
04 · Contracts and quality
Shape, range, freshness and referential expectations declared as tests that run on every load. A contract that only fails in a dashboard nobody opens is not a contract, so a breach either blocks the load or pages an owner.
05 · Semantic layer
Metric definitions live in one place with an owner, consumed by every tool rather than reimplemented in each. The alternative is the same number computed four ways in four tools, which is the state most teams are in when they ask us about AI.
Deliverables
What you are left holding.
Pipelines as code
Version-controlled, reviewed, deployed through your existing CI. No notebooks in production and no manually configured jobs.
Observability
Per-stage metrics, lineage, and quality alerts wired into the tooling your team already watches.
A tested recovery path
Backfill and rollback procedures that have been executed at least once against real data before handover, not documented and hoped for.
Capacity and cost model
What the system costs today, what it costs at three times the volume, and which knob moves that number most.
The hard parts
The questions worth asking us.
If a supplier cannot answer these specifically, they have not shipped one of these systems.
How do you migrate without a freeze?
Parallel running with reconciliation. The new platform is built alongside the old, both are loaded, and the outputs are compared row by row on an agreed set of measures until every difference is explained rather than merely small. Cutover happens per consumer, not per platform, so a failure affects one report rather than all of them.
What happens when a source system changes underneath you?
Schema changes are detected on ingest and fail the load rather than propagating. That sounds obvious and is rare, because the tempting alternative, coercing the new shape into the old one, keeps every dashboard green while the numbers quietly stop meaning what they meant.
How do you decide what to model first?
By what a decision depends on, not by what is easiest to load. The survey usually shows two or three subject areas carrying most of the questions. Those get modelled properly, and the rest lands raw and waits for a reason to exist.
Who owns it after you leave?
Your team, and we treat that as an engineering requirement rather than a handover meeting. The transformation layer, the tests and the deployment pipeline are the ones your engineers change during the engagement, not after it.
Related engagement
How this looks in practice.
Financial services
Replatforming a twelve-year-old warehouse without a reporting freeze
A migration run as two platforms in parallel for its whole duration, with a data contract on every source feed and cutover taken one reporting consumer at a time rather than one platform at a time.
UK building society, ~3,400 staff · 34 weeks
Read the engagementUsually engaged alongside
Start here
Talk to us about data & platform engineering.
Bring the estate, the constraint and the question you want answered. Forty-five minutes is usually enough to tell whether this is viable, and we would rather say so early.
- Prefer email
- hello@vectisflow.com
- Response time
- One working day, from a person who has read it.