Transforming Data Quality Assurance Through Automation

Transforming Data Quality Assurance Through Automation

Case Studies

My name is Priyanka Mishra, Principal Quality Assurance (QA) Consultant at Telefónica Tech within the Data & AI practice. On this project, I worked with a global legal organisation operating in a highly regulated, data-driven environment, supporting a critical transformation of their financial reporting platform. As Automation Test Lead, I was responsible for designing and building a data validation framework to ensure the integrity and trustworthiness of their data throughout the migration. This covered the full lifecycle: architecture design, Python development in Databricks, as well as the Azure DevOps pipeline orchestration. Alongside the technical delivery, I worked closely with stakeholders to ensure the solution aligned with business needs and delivered meaningful, reliable outputs.

About the project

Business professionals collaborating using Databricks Genie for data driven decisions

This project transformed how financial data quality was validated during a critical migration from Azure Analysis Services to Power BI, replacing manual, error-prone checks with a fully automated framework that delivers results in minutes. It aimed to deliver clear, actionable discrepancy reports that business analysts could interpret without technical support, while embedding validation directly into the CI/CD pipeline so data quality would become a mandatory deployment checkpoint.

With hundreds of financial measures underpinning critical business decisions, there was a non‑negotiable requirement to prove that every number in the new Power BI environment matched the legacy AAS system exactly. Any discrepancy, even a fraction of a percent on a revenue figure, could erode trust in the new platform and delay adoption.

The challenge

The customer’s move from Azure Analysis Services (AAS) to Power BI introduced a critical requirement: ensuring both systems produced consistent results despite differences in infrastructure, caching, aggregation, and refresh behaviour. The key challenges included:

Lack of scalability

Without automation, the team had no scalable approach and faced manually comparing hundreds of measures across multiple filter combinations, resulting in a slow, inconsistent, and error-prone process.

High risk to trust and adoption

Any discrepancies between AAS and Power BI risked undermining confidence in financial reporting and delaying adoption of the new platform.

Growing complexity

The challenge was compounded by over 200 measures, differing year-period mappings between systems, and the need to maintain alignment across multiple Power BI workspaces.

Limitations of manual checks

Earlier manual, side-by-side validation methods were difficult to repeat, covered only a limited set of scenarios, provided no audit trail, and offered no safeguard against future errors.

01 Establish the Approach

The solution was built on a simple idea: pull data from both systems, compare the results, and clearly highlight where they aligned and where they didn’t, turning complex validation into transparent insight.

02 Design for Users

Two distinct audiences shaped the approach. Developers needed detailed test outputs in Azure DevOps, while business users required clear, actionable Excel reports that were easy to interpret.

03 Build a Maintainable Framework

A three-layer architecture, Configuration, Validation, and Orchestration, was created to keep the solution powerful yet easy for the wider team to maintain and extend.

04 End-to-End Development with Stakeholder Input

The framework was fully developed using Python in Databricks and automated via an Azure DevOps YAML pipeline, with close collaboration to refine Excel outputs into clear, user-friendly reports.

05 Knowledge Transfer to Production

Technical sessions, structured handover, and careful planning ensured the team could take ownership and transition the solution smoothly and safely into production.

The Outcome

The project delivered a complete, automated framework that now runs seamlessly across four different environments in Azure DevOps. It carries out two major types of validation:

AAS vs Power BI checks – allowing any number of financial measures to be compared across multiple years and filters.
Power BI vs Power BI checks – making sure measures stay consistent between workspaces as changes are rolled out.

The framework generates clear, multi‑sheet Excel reports, including a high‑level summary dashboard and detailed tabs showing exactly where mismatches occur. Thanks to the user‑friendly configuration layer, business testers can add new validation tests themselves without requiring developer involvement.

The automated process now runs in about 10 minutes, transforming what was once days of manual effort.

Reflection

Reflecting on this project, I’m most proud of the architecture. Separating configuration from logic, keeping the comparison engine independent, and creating a simple interface for business users were all deliberate design choices and they proved their value throughout the project. When we introduced the Power BI vs Power BI checks, the core engine remained completely untouched. And when testers need to add new measures, they can do so themselves without ever writing Python.

If I were to approach the project again, I would implement robust error handling and logging from day one. We lost time early on to silent failures and vague error messages, and adopting stricter exception handling upfront would have made debugging far more efficient.