Skip to main content
VectisFlow
VectisFlow
All insights

Data modelling · 6 min

Slowly changing dimensions are a per-attribute decision

Tracking history on everything is not caution. It is how a warehouse becomes something analysts route around.

Platform engineering ·

Ask how a dimension handles change and you will usually be given a policy rather than a decision. Either the warehouse tracks history on everything, or it tracks it on nothing, and both answers were settled long before anyone looked at the attributes.

The policy is the problem. History is a property of an attribute, not of a table, and a policy applied to a table is a decision made in the wrong unit.

What blanket history costs

Take a customer dimension with forty attributes, all versioned. An email address changes and the row splits. A marketing preference flips and it splits again. Within a year there are ten rows per customer, and every query that wants the current value has to know to filter for it.

Analysts learn the filter, and then one of them forgets it once. The resulting number is wrong in a way that looks entirely plausible, which is the expensive kind of wrong. More often they avoid the dimension and rebuild the join in a spreadsheet, at which point the model has failed at the only job it had.

The test to apply, attribute by attribute

The question is not whether an attribute changes. Almost all of them do. The question is whether anyone will ever ask something whose answer depends on what the value used to be.

That question has a different answer for a risk grade than for a display name, and answering it four times for forty attributes is an afternoon of work that a table-level policy spends years paying for.

  • Grouped or filtered by, as at a past date: version it, this is what history is for
  • Current label only, contact details, display text, free-text notes: overwrite it
  • Changes on a known cycle, tariff, pricing band, risk grade: an effective-dated table of its own, rather than widening the dimension
  • Not queried by anyone in a year: ask whether it belongs in the dimension at all

Write the decision down beside the model

Per-attribute decisions are harder to remember than policies, which is the real reason policies win. Record the grain, the tracking mode and the reason in the model definition, in the same repository as the code that builds it.

Then the next engineer to touch the attribute can see that history was declined deliberately, and the argument is had once rather than every quarter. A decision nobody wrote down is indistinguishable from an accident six months later.

Versioning everything looks like the careful choice, and it buys something real: you never have to think about an attribute again. The bill for not thinking arrives as a dimension nobody wants to query, and it arrives late enough that the connection is rarely made.

Related

  • Data quality · 5 min

    A contract that cannot stop a load is not a contract

    A test that turns a tile amber and nothing else is documentation. A breach has to block a load or page an owner.

    Platform engineering ·

    Read
  • Evaluation · 6 min

    Your evaluation set is too big, and too small

    Two hundred stratified questions will find regressions ten thousand unstratified ones will miss. Coverage is a shape, not a count.

    Evaluation and assurance ·

    Read

Get in touch

Talk to us.

A first conversation runs about forty-five minutes and covers three things: what your estate actually looks like, whether anyone can define a correct answer or an agreed number, and whether your permission model resolves per user. Any one of them can rule the work out, and we would rather tell you in week one.

Response time
One working day, from a person who has read it.