Your Software Is Agile. Your Data Isn't: Closing The Gap Before AI Makes It Expensive

data governance

Enterprises have industrialized software delivery. Data quality is still assessed in the old way, after something has already gone wrong. Most organizations running on the Scaled Agile Framework SAFe) have software delivery down to a fine art. Automated testing, CI/CD pipelines, Agile release trains, built-in quality baked into every sprint – teams ship faster and with more confidence than they did five years ago.

Ask the same organizations how confident they are in their data, and the tone changes. Suddenly it's caveats: "mostly," "we think so," "depends which system you mean." Data quality issues get discovered in a regulatory return, a customer complaint, or a war room after go-live – never in a test environment, because there rarely was one.

This isn't a people problem. It's a gap in the model.

What we're seeing

  • Most data incidents in enterprise environments are found downstream – in production, a regulator's hands, or a customer complaint, not in test.
  • Organizations with 80-90% automated software testing often have less than 20% on the data layer feeding those same systems.

The cost is real. Gartner estimates poor data quality costs organizations an average of $12.9 million a year through flawed decisions, compliance exposure, and rework.

Why data keeps slipping through

Software testing has clear boundaries: a function, a screen, a release. Data doesn’t. It changes every hour, and it crosses team and system boundaries that rarely map onto a single delivery team. A defect in a table doesn’t fail a pipeline the way a broken function fails a build – it sits there, quietly wrong, until someone downstream notices the numbers don’t add up.

Three things compound the problem:

  1. A widening skills gap: Testers who excel at API contract testing but have never run a reconciliation.
  2. Tooling sprawl: A profiler from one vendor, a catalogue from another, neither talking to the other.
  3. A rising regulatory bar: UK GDPR, the Data (Use and Access) Act 2025,FCA Consumer Duty, the EU AI Act increasingly expect organizations to evidence data quality, not simply assert it.

"Test everything the way we test code" was never going to work for data. It needs its own discipline.

The real risk: AI doesn't fix this, it amplifies it

As organizations push machine learning and generative AI into production, poor-quality data stops being a back-office problem and gets automated at scale. A defect in a report is embarrassing and fixable. A defect baked into the training data behind a credit score, a triage decision or a pricing model compound silently, at the speed the model runs. The Information Commissioner’s Office (ICO) AI guidance and the FCA’s Consumer Duty point the same way: firms, especially in financial services, must prove their data-driven outcomes are fair; not just believe they are.

On a recent engagement, we found potential PII exposure in two places: the underlying data model and the reports built on top of it. Because sensitive attributes flowed through the model itself, fixing individual reports wouldn’t have closed the gap – any new report would have inherited the same risk.

We recommend fixing it at the source using Unity Catalog in Databricks.

  • Tagged PII centrally, creating one consistent record of where sensitive data lived.
  • Attached masking policies to the tags, so protection applied automatically wherever the data was accessed.
  • Established end-to-end lineage, from bronze to silver to gold, through to AI models and BI reports, for impact analysis and faster root-cause identification.

The protection now travels with the data: less human error, a consistent and auditable approach to PII, and a model safe to reuse. It also reflects a wider shift – governance moving from policy documents to controls the platform enforces automatically.

What good looks like

Organizations getting ahead of this treat data assurance as an extension of the built-in quality discipline they already apply to code, adapted to how data behaves. Four things run in parallel, not in sequence:

  • Testing the seams – where data moves between systems, which is where real defects usually live.
  • Automating reconciliation and validation at the speed data moves, not quarterly.
  • Treating the data as a test subject – profiling, baselining, watching for drift.
  • Keeping a human in the loop – for calls no anomaly detector can make: is this 0.4% a rounding artefact, or a six-figure exposure?

Six initiatives, one connected fabric:

Coverage should span the full data lifecycle. Three of the six show clearly what breaks without them.

  1. Safe, realistic test data. A payments pipeline passed every regression cycle on a masked production sample. Then a customer with no surname hit the live system and the transformation job silently dropped the record; the sample had been filtered into “clean” rows. Synthetic generators now cover what production rarely contains – nulls, mononyms, multi-byte characters, leap-day dates, duplicate keys – so the same defect fails in test, not in front of a customer.    
  2. Quality profiling for baselines and drift. An upstream team changed a currency field from pounds to pence. Schema checks passed because the column was still numeric, and revenue dashboards showed a 100x jump for two days until a finance analyst questioned the figures. Profiling now baselines each critical column – distribution, null rate, min/max, cardinality – and a shift in average order value alerts within the same load cycle.
  3. Data catalogues and metadata management. Marketing defined an “active customer” as a purchase in the last 12 months; support used it to mean a login in the last 90 days, and that definition reached a shared dataset. Two reports using the same term disagreed by 30%, and it took weeks to establish that both were right, and nobody owned the answer. The catalogue now holds an owner, a versioned definition and lineage, and any change triggers review of every dependent report.

The other three complete the picture:

  • Automated validation on every load: completeness, accuracy, consistency and business rules (“an order cannot ship before it is placed”) run as executable tests.
  • Embedded governance: AI-augmented classification and masking of sensitive data, especially in non-production, where controls tend to be looser.
  • Access control tested like code: RBAC and ABAC policies get positive and negative test cases and are re-tested every release, because silence is not evidence of correct access.

These aren’t six initiatives to run in isolation. They work best as one connected fabric: synthetic test data generated using the same classification rules that protect production; reconciliation exceptions feeding the catalogue’s lineage view, so a broken transformation is visible where an analyst would look. Run as six disconnected tools by six teams, each layer quietly undermines the other five.

Where to start

None of this needs one enormous program. Pick one Agile Release Train wrestling with a genuine data headache – a migration that’s harder to trust than expected, a domain three teams think they own – and prove the model there. Accelerators such as COCO can help build these checks into the delivery pipeline you already run. Make data quality visible where the train already looks, like the System Demo, not in a governance report nobody reads. Catalogue the highest-risk domains first, where an undocumented definition change costs the most.

The bottom line

Enterprises that have industrialized software delivery already know how to build quality in rather than inspect it in afterwards. The next step is applying that instinct to data, before AI adoption makes the gap between “agile software” and “untested data” too expensive to ignore.

Related articles

No items found.
Technology
Text Link