Applying Agile Software Engineering Principles to Data Governance

Applying Agile Software Engineering Principles to Data Governance Initiate Systems Business Management Consulting Practice Marty Moseley Chief Technology Officer Initiate, an IBM Company 1 © 2009 Initiate Systems, Inc.

About Your Presenter ê The way I think & operate (according to “Now Discover Your Strengths): 26+ years in IT-related technologies ê Enterprise Architect, Chief Technology Architect, Information Architect ê Entrepreneur & CTO – 4 startup companies, own consulting firm (~25 clients) ê Been working on CDI-kinds of problems for over 20 years 2 Connectedness Futuristic Ideation Intellection Strategic Telescope in one hand… …Wine Glass in the other © 2009 Initiate Systems, Inc.

Agile Data Governance Business Context Overview: Agile – Why “The New Black” Cherry-Picking some Agile DG Concepts – – – Forming a Lean DG Organization Building an Agile DG Business Case Prioritizing DG Projects Performing an Agile DG Project Evaluating Results and Value Lessons Learned Q&A 3 © 2009 Initiate Systems, Inc.

Level-Setting & Business Context “We find that clients that have formal data governance & management in place are those that can respond nimbly to market forces. ” – Jill Dyché 2009 4 © 2009 Initiate Systems, Inc.

Pressures Facing Our Organizations “Outside-In” Pressure Governance, Regulatory & Compliance pressures, security, privacy Global shifts in economic bases – China, India, APAC, Middle East, Africa Threats, fraud, malicious attacks “Inside-Out” Pressure Demand for real-time and continuous availability is going up Explosion in data volumes – in 2007 IDC predicted a sixfold increase thru 2010 – structured & unstructured Acquisitions, consolidation, restructuring within most sectors Number of systems and “moving parts” increasing IT must do more… with less Credit crises, market irregularities, corporate valuation fluctuations, revenue & competitive pressures Consumer & partner demands for online presence, transaction support, for online supply chains Technology advances such as SOA, Virtualization, Grid Computing Organizations are under increased pressure to make better decisions, in less time and with less risk, than ever before 5 © 2009 Initiate Systems, Inc.

What is Data Governance? Data Governance… An ongoing, evolutionary practice Business leaders define the principles, policies, processes, business rules & metrics required to meet business imperatives Data Governance … DG Board marshals people, organizations, resources, priorities and technologies They monitor, measure, & ensure accurate data are ‘fit for purpose’ – available when, where & how needed to meet business objectives 6 …an Ongoing, Evolutionary Practice © 2009 Initiate Systems, Inc.

Why Data Governance? There are some organizational problems that are directly caused by poor data quality – Privacy violations & Security breaches – Fraud & abuse due to lax data practices – Poor Consumer/Citizen satisfaction and loyalty due to mismanaged personal, household, and company data – Waste and rising expenses due to duplicate, unmanaged data There are some opportunities that go unrealized because poor data quality stands in the way – Opportunities to provide additional services – Opportunities to increase participant loyalty – Opportunities to lower costs or increase revenue These are always changing – With Organizational, Mandate, Mission changes, IT upgrades 7 © 2009 Initiate Systems, Inc.

So, How are we Doing? 50% 75% …of organizations say data is ‘worse than everyone thinks’. Source: TDWI (2006) 74% …of organizations don’t have metrics to measure data quality. Source: Principia (2006) 8 81% …‘inconsistent data definitions’ & ‘data entry’ are source of data quality problems. …of BI/DW initiatives suffer due to the poor quality of master data. Source: TDWI (2006) 41% …of organizations have no clearly defined owner responsible for data governance. Source: TID (2008) 52% …of organizations have not attempted to calculate the cost of poor data. Source: TID (2008) Source: TDWI (2007) 61% …of organizations do not measure DQ. Source: TID (2008) © 2009 Initiate Systems, Inc.

Agile Fundamentals – The New Black The “new black” means that we’ve seen an impressive uptake in Agile/Lean methods and those same techniques can successfully be applied to Data Governance and Data Stewardship initiatives. 9 © 2009 Initiate Systems, Inc.

The Agile Approach to Software Development Built on the concept of iterations – Smaller, focused teams work more efficiently, build cohesiveness – Leverages well-defined roles – Continuous builds of working, tested code enables early value confirmation – Enables learning, refinement, improvement, tuning – Creates predictability, easier ROI Attacks the problem of complexity Focuses on solving one bite-sized chunk (problem) at a time Build “good enough” solutions – not “solve world hunger” efforts that creep ever larger Teams stay focused; they don’t get distracted by goals that are too big, too lofty, too far-reaching, and too obtuse to see business benefit and value 10 © 2009 Initiate Systems, Inc.

The Agile vs Traditional Approach to DG 11 © 2009 Initiate Systems, Inc.

The Traditional Approach to DG Appropriately involves executive management as executive sponsors, but without appropriate constraints – Months of time wasted in discussing roles & responsibilities, team makeup, goals and objectives Similar in mindset to “waterfall” approaches to SW development – Plodding; early deliverables no visible value Tendency to “boil the ocean” and solve every data problem Ends up failing due to the sheer size of the effort & the inordinately long time before progress is made Executives are too busy for this and see this as a waste of time – They are rewarded for “making their numbers” and do not see a correlation of DG to “growing their business” 12 © 2009 Initiate Systems, Inc.

The Agile Approach to DG Laser focus on solving one problem at a time – Relies on execs to prioritize, direct, based on value to the business Built on the concept of iterations similar values as Agile SW Dev. – Shorter time-frames enable line-ofsight between tasks and end goals, ensure that measurable progress – Enables learning, refinement/ improvement, tuning – Creates predictability, easier ROI Executives stay focused; they don’t get distracted by goals that are too lofty, too far-reaching, and too obtuse to see business benefit and value 13 © 2009 Initiate Systems, Inc.

Agile Data Governance: An Iterative, Adaptive DG Process 1. 2. Form the Data Governance Board Define the Problem & the Team 7. 3. …a series of independent, focused sprints that help prioritize problems, develop ‘good enough’ solutions, learn & improve, and repeat Evaluate Progress & Plan Next Steps 6. Nail Down Size & Scope 4. 5. Implement the Data Quality Solution Validate Your Assumptions Establish Data Policies 14 © 2009 Initiate Systems, Inc.

DG & MDS Project Alignment – Great Synergy 1. Form DG Board 2. Define Problem & Assign Team 3. Nail Down Size & Scope Validate DQ Findings 4. 5. Establish Data Policies 6. Implement the Solution 7. Evaluate Progress & Next Steps Task/process alignment between a DG project and an Initiate MDM project 1. Initiation (Kick Off Project) 2. Configuration (Detailed Analysis) 3. Configuration (Create MDS) 4. Configuration (implement Policies) 5. Verification (Test & Tune MDS) 6. Deploy MDS 15 © 2009 Initiate Systems, Inc.

Step 1: Data Governance Board 1. 2. 3. Form the Data Governance Board Define the Problem & the Team Nail Down Size & Scope 4. 5. Validate Your Assumptions 6. 7. Establishing Data Policies Implement the Data Quality Solution Evaluate Progress & Plan Next Steps Small guidance team of executives from across IT System the ecosystem Owners Can think cross-organi. Executive Sponsor zationally and prioritize cross-functional opportunities Operations Agrees on top n data quality priorities Software Establish priorities & ground rules QA Mobilize resources on DG program increment CIO Plays Advisory Role Direction-setting, Leadership, Momentum 16 Data Owner Analyze Opportunities, Establish Priorities, Clear Roadblocks Business Priorities, Gaps, Risks, Issues, Opportunities, Tradeoffs Biz Process Owners Enterprise Architects Data Governance Board Integration Architects Data Stewards Software Engineers © 2009 Initiate Systems, Inc.

Step 2: Defining the Problem & the Team 1. 2. 3. Form the Data Governance Board Define the Problem & the Team Nail Down Size & Scope 17 4. Validate Your Assumptions 5. 6. 7. Establishing Data Policies Implement the Data Quality Solution Evaluate Progress & Plan Next Steps Examine issues, root causes, Represent Application Constraints & Conflicts initial scope in participating IT System systems/ BUs, risks with Evangelist, Owners Spokesperson, selected increment Herder-of-Big-Cats Define & Prioritize Business Executive Sponsor Drivers Appoint resources to Increment Operations team (across IT and necessary BUs) Software During Increment, run QA interference, remove roadblocks, approve Policies, Processes, Business Rules & Metrics Agree to Take Ownership of Data Identified Data Owner Find Opportunities, Ensure Exec Participation Assign Priorities Represent Business Needs, Gaps, Risks, & Opportunities Biz Process Owners Enterprise Architects Data Governance Board Integration Architects Data Stewards Software Engineers © 2009 Initiate Systems, Inc.

Define a Simple Business Case Customer Experience – Cost of Dissatisfaction; Loyalty & Retention; KYC Productivity – Effectiveness of Sales, Marketing, BI; Resource Alloc. Pricing & Discounting – Tailored pricing based on detailed Customer knowledge Customer Profitability & Value Analysis – 360° Customer knowledge enables value assessment Cost of Sales – Territory Alignment; Service Eligibility; Credit Risk Financials – Compensation Fidelity; Financial Reporting Compliance – SOX, BASEL II, etc; Privacy Requirements 18 © 2009 Initiate Systems, Inc.

Assigning Value or Cost to Data Quality How Does DQ Contribute to: Reduced risk? Reduced cycle times? Reduced cost of doing business? Examples: Commercial KYC Fraud Detection Marketing Effectiveness Customer Satisfaction Public Sector Increased revenue? Increased safety? Protection from Terrorism Fraud Prevention Healthcare Increased satisfaction? Can you measure these? 19 Protecting patients’ lives, Sharing medical records, Prevent fraud © 2009 Initiate Systems, Inc.

The Big Q: What Should You Govern & Why? Customers? Products? Locations? Accounts? Suppliers? Contacts? Portfolios? Contracts? Counterparties? Assets? Licenses? Securities? Calendars? Organizations? 20 MARCH JOHN Q. PUBLIC © 2009 Initiate Systems, Inc.

Step 3: Nail Down Size & Scope 1. 2. 3. Form the Data Governance Board Define the Problem & the Team Nail Down Size & Scope 4. Validate Your Assumptions Focus & Prioritize!! Examine in detail the scope of the Increment: doable in <9 months? Involve leaders/owners from the business and IT Do NOT strive for 100% Limit Scope Creep – examine # of systems, functional areas, transactions supported, feeds 6. 7. Establishing Data Policies Implement the Data Quality Solution Evaluate Progress & Plan Next Steps Represent Application Constraints, Changes, Grant Access to Systems Evangelist, Driver, Spokesperson 21 5. IT System Owners Executive Sponsor Drive Scoping Size Data Owner Clear Roadblocks, Drive Analysis, Get Access Represent Business Needs & Value of DQ Changes Biz Process Owners Enterprise Architects Data Governance Board Integration Architects Operations Software QA Data Stewards Software Engineers Search for Extent of Bad Data; Size the Project © 2009 Initiate Systems, Inc.

Step 4: Validate Your Assumptions 1. 2. 3. Form the Data Governance Board Define the Problem & the Team Nail Down Size & Scope 4. Validate Your Assumptions Challenge all assumptions & Estimates (scope, time, resources…) Let the Data Stewards and analysts loose to examine the quality of source data Use tools like profiling and probabilistic matching to examine data quality Outcome: List of DQ issues that need to be solved and why they must be resolved 6. 7. Establishing Data Policies Implement the Data Quality Solution Evaluate Progress & Plan Next Steps Provide Access to Data from Systems Evangelist, Spokesperson 22 5. IT System Owners Executive Sponsor Weigh & Prioritize Problems, Open Doors to Business Leaders Data Owner Clear Roadblocks, Open Doors Represent Business Needs Biz Process Owners Enterprise Architects Data Governance Board Integration Architects Operations Software QA Data Stewards Software Engineers Test the Data from as Large a Sample as Possible © 2009 Initiate Systems, Inc.

Step 5: Establish Data Policies, Processes, Rules & Metrics 1. 2. 3. Form the Data Governance Board Define the Problem & the Team Nail Down Size & Scope 4. Policies cover all dimensions of quality: completeness, consistency, accuracy, security, fitness Policies include ownership, roles, & responsibilities Policies cover exception management processes Also look at rules to ensure they don’t break sharability Metrics – less is more! Business metrics are better than technical metrics! Validate Your Assumptions 6. 7. Establishing Data Policies Implement the Data Quality Solution Evaluate Progress & Plan Next Steps Represent Systems’ Abilities to Enforce Policies Provides Business Relevance; Spokesperson 23 5. IT System Owners Executive Sponsor Write & Approve Policies Data Owner Approve Policies Represent Business Process Needs, Policy Compliance Biz Process Owners Represent Infrastructure by Policies Enterprise Architects Data Governance Board Integration Architects Operations Software QA Data Stewards Software Engineers Draft & Test Policies © 2009 Initiate Systems, Inc.

Principles, Policies, Processes, Rules & Metrics Definitions Principles The highest-level statements of purpose, values, mission, behaviors, and mores; statements of “why” things must be achieved – “philosophical touch-stones’ (Jill’s quote) Policies Measurable statements of “what must be achieved” for which kinds of data, who owns responsibility, how disputes are settled Processes Discrete tasks and procedures required to meet policies’ objectives Business Rules Defines “how” the policies’ objectives are met; how anomalies/failures/exceptions are handled Metrics 24 The “critical few” metadata facts captured about data transformations that provide insight into how tasks are running © 2009 Initiate Systems, Inc.

Principles, Policies, Capabilities, Processes & Rules Example Principle Policy The Customer is always right. Therefore, we will strive to always delight the Customer in every interaction. Whenever a Customer is encountered and we capture data about them, the capturing application will enforce Minimum Data Requirements. The CMO owns the definition and rules for Customer. The only allowed exceptions are {…}. Exceptions to this policy will not cause a transaction termination {exceptions being…}, but will cause a “flag and notify” response to… The following metrics will be captured to measure if this policy is being enforced {data source, transaction type, date/time of transaction, transaction ID, value, exception data…} Process Customer exceptions must invoke {…} remediation process Customer exceptions must invoke {…} acceptance process Customer exceptions must invoke {…} audit process Business Rule Customer Name will follow the ______ Domain standard {…} If exceptions are raised, then perform the following tasks… Metric 25 Whenever a duplicate record is suspected, capture the context and report the anomaly Whenever a record is updated and evaluated for a match, record the event Whenever a message is rejected because the schema was the wrong version, record the event details © 2009 Initiate Systems, Inc.

Some Principles About Policies þ Less is More – not everything needs a policy – follow the 80/20 rule þ Examples are good þ Policies should state ‘business’ principles more than system and program/ initiative names that will change over time þ Policies have to be ‘actionable’ – if they can’t be implemented, measured, and managed, they won’t work þ Policies should not be ‘shelfware’ – if they’re not living documents they won’t get used (Wiki’s are ideal for creating, evolving, and communicating data quality policies) þ Context is key: Although other functional areas are out of scope, take a quick look to ensure the policies and rules won’t break anything in future increments þ Reuse industry standards for reference data quality – don’t reinvent the wheel! þ Only mandate what’s broken þ An Iteration should not produce more than 20 -ish policies 26 © 2009 Initiate Systems, Inc.

Step 6: Implement the Data Quality Solution 1. 2. 3. Form the Data Governance Board Define the Problem & the Team Nail Down Size & Scope All hands on deck! Architect the Solution for short- and long-term applicability This is where scope management pays off Policies, processes, business rules & metrics tied to system assets (services, messages, schemas, feeds, user tools) 27 4. 5. Validate Your Assumptions 6. 7. Establishing Data Policies Implement the Data Quality Solution Evaluate Progress & Plan Next Steps Represent Application Constraints, Changes, Grant Access to Systems Evangelist, Spokesperson IT System Owners Executive Sponsor Manage Policies, Manage DQ, Own Responsibility for DQ Data Owner Analyze Priorities, Clear Roadblocks, Approve Policies Represent Business Process Needs, Policy Compliance Biz Process Owners Oversee Overall Solution Design Enterprise Architects Data Governance Board Integration Architects Operations Ensure Quality of Implementation Software QA Test the Implementation Data Stewards Software Engineers Ensure Integration Compliance Draft & Test the Policies Implement the Policies © 2009 Initiate Systems, Inc.

Step 7: Evaluate Progress & Plan Next Steps 1. 2. 3. Form the Data Governance Board Define the Problem & the Team Nail Down Size & Scope 4. Validate Your Assumptions 5. 6. 7. Establishing Data Policies Implement the Data Quality Solution Evaluate Progress & Plan Next Steps Evaluate Costs, Measures & Metrics collected during Benefits to System Assets the Increment (forecast vs actual IT System scope, HC, deliverables, etc. ) Owners Evangelist, Spokesperson How many policies, processes, etc. ? Executive Sponsor Did you achieve the value you desired? Operations Review what worked & learn from mistakes - remember to Capture Costs of Implementation Software blame the process, not people! QA Plan next DG/DQ Project – will you do anything differently? 28 Publish Findings, Capture Metrics, Ensure Compliance Data Owner Approve Metrics & ROI; Re-evaluate Priorities for Next Increment Evaluate Compliance & Benefits of the Policies Biz Process Owners Monitor & Measure Solution Design Enterprise Architects Data Governance Board Integration Architects Data Stewards Software Engineers Run Spot Checks to Measure Data Quality Results © 2009 Initiate Systems, Inc.

Approaches That Work: Recommendations from Hard-Learned Lessons “Six months of accelerated data remediation is cheaper than 3 years of maintenance. ” – Saskatchewan 2009 29 © 2009 Initiate Systems, Inc.

Lessons Learned & Best Practices o o o o o 30 Iterate! Show real value in <9 months Involve business leaders to begin Data Governance Leverage SOA & MDM together for the short and long term success of your data quality initiatives Architectural Styles – Hybrid is the best ownership model Hub & Spoke – The right architecture (follows SOA principles) Plan for multi-domain master data management Use SOA to enforce data quality at the point of entry, not after transactions have been committed Quality is everything – Don’t sacrifice accuracy for performance – you can definitely have both © 2009 Initiate Systems, Inc.

What NOT To Do û Don’t spend time trying to create the perfect data model of your entire organization û Don’t think of this from a Data Warehouse perspective o o – One big DB with everything in it – Batch-oriented think you’ll get everything û Don’tof breed is the best you canyou need from one vendor – best get for the foreseeable future in production in û If you can’t get something to do too much 6 -8 months, chances are you’re trying scalable DB and matching engine yourself û Don’t try building thethink and you’ll spend multiples of what – it’s harder than you o o o û o 31 it’ll cost you to buy a solution Don’t try to build an enterprise solution from something that only scales to a departmental level © 2009 Initiate Systems, Inc.

Initiate, an IBM Company We enable organizations to strategically leverage and share critical data assets Leading pure-play CDI/MDM vendor Established 1995 Cited as ‘leader’ by Gartner and Forrester 200+ customers across 10 market segments – Private and Public Healthcare, Healthcare Exchanges – Finance, Retail, Hospitality, High Tech, Manufacturing – State and Local Government, Intelligence, Law Enforcement Offices in U. S. , Canada, EMEA, Asia-Pacific Follow Us on: blog. Initiate. com 32 “Initiate Syste ms productive live …the most customers. ” time vers fast low deli Initiate…nd a relatively tio” “ a a to value -software cost r o service-t © 2009 Initiate Systems, Inc.

Initiate Business Management Consulting We provide consulting in several areas to help you with Data Governance and MDM: MDM Business Consulting Services Data Governance Consulting Services MDM Business Case: 3 -4 week engagement – part of what we’ve offered to date as part of MDM Assessments DG Readiness Assessment: 2 -3 week assessment of an organizational ability to operate a DG initiative MDM Strategy, Architecture & Roadmap: 3 -4 week engagement – part of MDM Assessment DG Business Case (99% the same as the MDM Business Case): 3 -4 week engagement – leverages the MDM Business Case MDM Data Quality Audit: 3 -6 week engagement where we would profile data selected for a target project Establishing the DG Board: 3 -6 week engagement creating a Data Governance board MDM Program Management: 6 -8 month project where we help the client run a Data Governance project that coincides with an implementation of our software Data Modeling: Its alignment with MDM good practices 33 DG Policy & Biz Rule Development: 6 -8 week engagement captures business principles, processes, policies, metrics and KPI’s that provide for continuous monitoring of MDM project performance DG Data Quality Assessment DG Program Management Data Stewardship: Organization, roles, and responsibilities © 2009 Initiate Systems, Inc.

Q&A Marty Moseley Chief Technology Officer 200 W. Madison, Suite 2300 Chicago, IL 60606 Mobile: +1 408 315 9572 Office: +1 800 992 0300 mmoseley@intiate. com You Can Also Follow Us on: Twitter 34 Facebook Linked. In blog. Initiate. com © 2009 Initiate Systems, Inc.