Virtual University of Pakistan Data Warehousing Lecture-23 Total

Virtual University of Pakistan Data Warehousing Lecture-23 Total DQM Ahsan Abdullah Assoc. Prof. & Head Center for Agro-Informatics Research www. nu. edu. pk/cairindex. asp National University of Computers & Emerging Sciences, Islamabad Email: ahsan 101@yahoo. com DWH-Ahsan Abdullah 1

Data Quality Management Process Establish TDQM Environment Evaluate Data Quality Management Methods Scope Data Quality Projects & Develop Implementation Plans Implement Data Quality Projects (Define, Measure, Analyze, Improve) 2

Data Quality Management Process 1. Establish Data Quality Management Sub-bullets will not go to graphics Environment • IS project managers • Development professionals. • Functional users of legacy information systems with domain knowledge • IS developers know solutions but don’t know how and where to modify 3

Data Quality Management Process yellow will go to graphics 2. Scope Data Quality Projects & Develop Implementation Plans • Task Summary: Project goals, scope, and potential benefits • Task Description: Describe data quality analysis tasks • Project Approach: Summarize tasks and tools used to provide a baseline of existing data quality • Schedule: Identify task start, completion dates, and project milestones • Resources: Include costs connected with tools acquisition, labor hours (by labor category), training, travel, and other direct and indirect costs 4

Data Quality Management Process 3. Implement Data Quality Projects (Define, Measure, Analyze, Improve) Yellow will go to graphics • Define: Identify functional user DQ requirements and establish DQ metrics • Measure: conformance to current business rules and develop exception reports • Analyze: Verify, validate, and assess poor DQ causes. Define improvement opportunities • Improve: Select/prioritize DQ improvement opportunities i. e. data entry procedures, updating data validation rules, and/or company data standards. 5

Data Quality Management Process 4. Evaluate Data Quality Management Methods • modifying or rejuvenating existing methods of DQ management Sub-bullets will not go to graphics • determining if DQ projects have helped to achieve demonstrable goals and benefits. Evaluating and assessing DQ work as, it is not a program, but a new way of doing business. 6

The House of Quality Matrix Technical Correlation Matrix Customer Requirements Technical Design Requirements Interrelationship Matrix DWH-Ahsan Abdullah 7

How to improve Data Quality? The four categories of Data Quality Improvement § Process § System § Policy & Procedure § Data Design 8

Quality Management Maturity Grid CMM Level-1 Uncertainty CMM Level-2 Awakening CMM Level-3 Enlightenment CMM Level-4 Wisdom CMM Level-5 Certainity 9

Misconceptions on Data Quality § You Can Fix Data § Problem NOT in data, but how it was used. § It is NOT a one time process. § Buying a cleansing tool is NOT the solution. § Some live with the problem, cant afford the tool. Sub-bullets will not go to graphics § Data Quality is an IT Problem § It is the company problem. § Define the metrics of quality. § Business has to strike a balance between quality and ROI. § Joint business and IT effort. 10

Misconceptions on Data Quality § (All) Problem is in the Data Sources or Data Entry § NOT the only problem. § Systems could be responsible, but actually it is the metrics. § Two divisions using different codes for same entity. § Need to track, trace, check data from creation to usage. § The Data Warehouse will provide a single source of truth § In ideal world it is indeed true. § In real world maybe multiple data warehouses, data marts, external source i. e. silos of data resulting in multiple sources of “truth”. § Even with single source of truth, if transformations and interpretations are different, an issue. 11 Sub-bullets will not go to graphics

GIGO 12