Article
Nobody decides to build six copies of the same dataset. It happens one reasonable decision at a time — and the fix is governance, not a tidying exercise.
Nobody decides to build six copies of the same dataset. It happens one reasonable decision at a time.
A business unit needs a report. The data exists, but they cannot see all of it, or they can see too much of it, or the permissions question is genuinely complicated and the person who could answer it is busy. So somebody creates a dataset scoped to that team. It takes an afternoon. Everyone is happy.
Six months later another team needs something similar, and there is now a precedent. Eighteen months later you have several datasets built on the same underlying data elements, each serving one audience, each maintained separately, and a warehouse considerably larger than the information it contains.
This is a governance problem that has been solved by duplication, and it is worth recognizing because it is expensive in three different ways at once.
Governing access properly means deciding who should see what, encoding those rules, and maintaining them as people move between roles. It requires a decision from someone with authority, and it is the kind of work that is never urgent.
Copying a dataset requires a ticket.
Given the deadline, most organizations will take the ticket every time, and the person taking it is not making a mistake. They are solving the problem in front of them with the tool that works today. The cost lands later and lands somewhere else, which is exactly why this pattern is so durable.
Licenses. More instances means more of whatever your platform charges for. This is the visible cost and usually the smallest one.
Storage. Every copy of a dataset is a copy of the data. In a warehouse holding several versions of the same information, a meaningful share of what you are paying to store is redundancy.
Divergence, which is worse than either. The copies do not stay identical. One gets a filter adjusted, another a calculated column added, a third is refreshed on a different schedule. Then two business units bring numbers to the same meeting and the numbers do not agree, and the next hour goes on working out which is right rather than what to do about it.
That last one is the real cost, and it never appears on an invoice.
In one enterprise estate we inventoried more than a hundred datasets. Thirty-two were outright duplicates. Another twenty-eight were built on the same underlying metadata and collapsed into a single denormalized dataset once somebody looked at them side by side.
100+
datasets in the estate
32
outright duplicates
28
collapsed into one
40%
smaller warehouse
The storage number is the easy one to quote and the least interesting part of the exercise. Two things mattered more. The warehouse was also backed up, so every redundant copy was being paid for more than once and recovery was more complicated than it needed to be. And the consolidated estate was reorganized under a schema naming convention, so that the next time somebody needs a report, the naming tells them whether the data already exists.
That is the part that stops the problem recurring: discovery is now easier than duplication, which is the opposite of how it was.
It is tempting to treat this as a tidying exercise — merge the datasets, delete the duplicates, move on. That fails, because the reason the copies exist has not changed. Within a year you will have new ones.
The fix is to make the thing that copying was working around unnecessary. One governed dataset, with row-level security determining what each user sees. Each business unit gets its own view of the data without its own copy of the data.
How it worked in practice
The access model itself did not need reinventing. It was already organized by business unit, with each unit's membership defined by a security group — finance's group, and so on down the org chart.
In the duplicated world, that group decided which copy of the data you were allowed to open. In the consolidated one, the same group decides which rows you see. The organizational logic was never the problem; it was being enforced by making copies instead of by filtering one.
Row-level security is usually discussed as a governance control, and it is one. It is also a cost decision. When access is enforced at the row rather than by duplicating the dataset, the license count and the storage footprint both fall out of it.
They did for us, and the objection was reasonable.
When teams heard their datasets were being merged, or that they would be pointed at new dashboards, the concern was not architectural. It was that they would lose access to reports their business depends on. Nobody was defending a copy of a dataset. They were defending the report that runs on it, which is a different thing and a legitimate worry.
What resolved it was showing rather than reassuring. For each team, we walked through where their report's data comes from today, then through where it would come from after consolidation — the same data, fewer layers, arriving by a shorter path. Once a team could see their own report at both ends of the change, the conversation stopped being about whether to do it.
The argument that landed hardest was reliability, not cost. Fewer copies means fewer layers, and fewer layers means the number on the dashboard is more likely to be right.
Every copy has a happy owner. The business unit that had a dataset built for them is not experiencing a problem — they got what they asked for, it works, and they are not paying the storage bill.
The cost is distributed and the ownership is not. It sits with whoever is responsible for the platform as a whole, and in most organizations that person is not in the conversations where the copies get requested.
This is the part where an outside party has an advantage, and it is not expertise. It is standing. Consolidating a dataset means telling a team that the thing they were given is going away. That conversation is easier for someone whose job is the whole estate than for a colleague who has to work with them next week.
Three questions, in order of how uncomfortable the answers usually are
How many datasets do you have, and how many distinct source tables do they draw on?
A large gap between those numbers is the signal.
If two business units report the same measure, do the numbers match?
Check rather than assume.
Who decides when a new dataset is created?
If the answer is whoever asks, you already know what you will find.
The point
Duplicated datasets are not a data problem. They are what happens when governance is harder than copying, and they will keep happening until that is no longer true.
Fix the access model and the copies stop being necessary. Name things so the next person can find what already exists. The license saving and the smaller warehouse are consequences, not the objective — though they do tend to pay for the work.
Written by
Data & Analytics Lead, Modern Signal
Modern Signal's data and analytics practice, working across managed services engagements in regulated industries.
Where this came from
A healthcare provider
A healthcare provider's technology estate had grown by addition, with every business unit solving its own problem. We took over running it, then rebuilt what duplication had made expensive.
Read the case study: Entered once, not three times — 70% faster RCM workflows and 35% off licensing~70%
Faster revenue cycle workflows
Related
Tell us what it looks like today and we will tell you what we would look at first.
Nothing here is required except an email address and enough detail to reply usefully. The more you tell us, the better the first answer.
