Topics Data Warehousing Concept Data Access Technology Enterprise

Topics Data Warehousing Concept Data Access Technology Enterprise Real-Time Knowledge Architecture for Data Warehousing Data Collection and Delivery

Benson & Parker’s “Square Wheel” Technology Environment Business Planning Business Operations M. Anvari Page 3

Benson & Parker’s “Square Wheel” Technology Environment Business Planning Technology Planning Business Operations M. Anvari Technology Operations Page 4

Benson & Parker’s “Square Wheel” Technology Environment Business Planning Impact Organization Business Operations M. Anvari Technology Planning Opportunity Alignment Page 5 Technology Operations

Benson & Parker’s “Square Wheel” Technology Environment Business Planning Impact Technology Planning Information Technology has to do more than Organization Opportunity just align itself with the business, it has to help Alignment the business have the maximum Technology in the impact Business Operations marketplace. M. Anvari Page 6

Data Access and Delivery System

Technology Evolution l New classes of computers l New classes of communications l New classes of technology (image, sound, video, multimedia) l New classes of software l Much more complex technical environment n Cooperative Processing/Client-Server n Distributed Data Bases n LANs, WANs, etc. l Obsolescence Problem n M. Anvari Multiple Legacy Systems Page 8

IT Impact on Business Enterprise Network Computing and Client/Server Technology are changing the way organizations look at all of their information systems HP Compaq Ob s IT oles W ce as nc te e s DEC IBM D at a Ja il M. Anvari Page 9

The Existing Enterprise l Support Existing Products l Support Existing Customers l Support Existing Organization l Support Existing Workforce l Support Existing Technology M. Anvari Page 10

Controlling the (Global) Real-time Organization RTO = 24 x 7 x E (Where E means every major market) M. Anvari Page 11

Information and the Enterprise Organizational needs for data Organizational needs for information Organizational needs for knowledge M. Anvari Page 12

Needs for Data = Values (Measurements) l Data to operate l Data to control l Data to plan M. Anvari Page 14

Needs for Information = Content + Structure (Relationships) l Structure of the Real-world l Relating data to the business n Cross functional processes l Relating data to the real world n External DB n External Data Feeds (D&B, Reuters, etc. ) n Text, Image, Voice, Video, etc. n Statistical Studies M. Anvari Page 15

Needs for Knowledge = Goals + Actions + Learning l Learning more about our business l Learning more about our market l Learning more about the business environment Knowledge is the area in which Data Warehousing and Data Mining are potentially critical technologies M. Anvari Page 16

Data, Information and Knowledge l Data Centers l Data Bases l Information Centers l Information Bases l Knowledge Centers l Knowledge Bases M. Anvari Page 17

Old Data Never Dies 60 s 70 s 80 s 90 s Batch On-line Minis PCs Networking Enterprise Computing (Peer to Peer, Network to Network) Note that none of the early computing styles have ever gone away!!! M. Anvari Page 18

Operational vs. Informational Systems Information Access Today M. Anvari Page 19

Operational vs. Informational Systems Mafg. Operational Systems Ord. Entry Information Access Today M. Anvari Page 20

Operational vs. Informational Systems Operational Systems Information Access Today M. Anvari Page 21

Operational vs. Informational Systems Operational Systems Estimating & Analysis Marketing Systems Product Planning Informational Systems Information Access Today M. Anvari Page 22

Operational vs. Informational Systems Operational Systems Information Delivery System Informational Systems Information Access Today M. Anvari Page 23

Operational vs. Informational Systems Operational Systems Information Delivery System Data Warehousing is fundamentally an issue of Enterprise Data Informational Architecture Systems Information Access Today M. Anvari Page 24

Operational vs. Informational Systems Operational Systems Information Delivery System Informational Systems M. Anvari Page 25

Operational vs. Informational Systems Operational Systems Data Information Warehouse Delivery System Informational Systems M. Anvari Page 26

Operational vs. Informational Systems Operational Systems Data Information Warehouse Delivery System Data Marts Informational Systems M. Anvari Page 27

Operational vs. Informational Systems External Data Operational Systems Data Information Warehouse Delivery System Informational Systems Data Garages M. Anvari Page 28

Operational vs. Informational Systems External Data Operational Systems Data Information Warehouse Delivery System Informational Systems External Users M. Anvari Page 29

End User Evolution l Data Base Management Systems users l Ad Hoc Reports users l Today’s Customer Demands Automated Real-Time Response. l End User Systems n Decision Support Systems n Executive Information Systems n Information Centers M. Anvari Page 30

Ways to Organize Data l Tables Flexible, Simple l Hierarchies Speed, Natural Reporting l Multiple Directions, Complex Structure Networks l Lists Updating Complex Structure l Matrices / Array Manipulate Multiple Dimensions l Inverted Files Unplanned queries, text retrieval l Objects Complex structures, hide structure l Multidimensional Data Bases (Data Warehousing) M. Anvari Page 31

End User Computing Evolution M. Anvari Page 32

Data Warehousing Data Warehouse can be thought of as an automated version of the Information Center that was widely popular in the mid-1980 s or even ultimately as the automation of Information Resource Management. And while technologies such as client-server have begun to put enormous computing and graphics power in the hands of individuals, however, these technologies have not, in general, provided the link to the operational data that end users need to make critical business decisions. M. Anvari Page 33

Data Warehouse Requirements Support for Universal Access to Multi-platform Data Bases Support for Multiple User Types Separation of Operational and Informational Concerns Support for Networked Data Support for Directories, Repositories and Information Models, Support for Advanced End User Interfaces M. Anvari Page 34

Access to Heterogeneous Data HP Compaq DEC IBM M. Anvari Page 35

Multiple User Types (Knowledge workers) l l l l M. Anvari Top Executives Managers Analysts Planners Product Developers Consultants Lawyers etc. Page 36

Separation of Operational and Informational Concerns l Operational Systems n Response Time n Reliability n Security n Recoverability l Informational Systems n n Large numbers of different views n Manage Huge Amounts of Data (VLDBs) n Need to drill down/drill thru into data n M. Anvari Flexibility, Performance, Ease of Navigation Need to draw on data from many sources Page 37

Support for Networked Data All the data that is required to support informational needs is often not on the same operational data base. The need for Labor Negotiations, for example, may come from a variety of operational data bases, such as Manufacturing, Personnel, and Accounting. Distributed Systems M. Anvari Page 38

Support for Advanced End User Interfaces M. Anvari Page 39

Dimensions of Data Warehousing Performance Security Connection to the Operational Data Ease of Use Flexibility Distributed Data Quality Scalability M. Anvari Page 40

Enterprise Knowledge Architecture for Data Warehousing M. Anvari Page 41

Operational vs. Informational Systems Operational Systems Information Delivery System Informational Systems M. Anvari Page 42

Operational vs. Informational Systems M. Anvari Page 43

Enterprise Network Computer Architecture Data Mart M. Anvari Page 44

Freeing the “Data in Jail” M. Anvari Page 45

The Information Access Layer M. Anvari Page 46

The Legacy Data Layer M. Anvari Page 47

The External Data Layer M. Anvari Page 48

The Data Access Layer M. Anvari Page 49

The Data Access Layer Data Access Filter M. Anvari Page 50

The Data Access Layer SQL Queries M. Anvari Page 51

The Data Access Layer SQL Queries SQL Answers M. Anvari Page 52

Application Messaging M. Anvari Page 53

The Meta-Data Repository Layer M. Anvari Page 54

The Process Management Layer M. Anvari Page 55

The Core Data Warehouse M. Anvari Page 56

Data Staging and Quality M. Anvari Page 57

Data Mart (Post-process/Indexing) Post. Proc. & Indexing M. Anvari Page 58

Goals of Warehouse 1. Performance (Canned queries, MD Analysis, Ad hoc, Impact on Operational System) 2. Flexibility (MD Flex, Ad hoc, Change data structure) 3. Scalability (No. of Users, Volume of Data) 4. Ease of Use (Location, Formulation, Navigation, Manipulation) 5. Data Quality (Consistent, Correct, Timely, Integrated) 6. Connection to the Detail Business Transactions M. Anvari Page 59

Virtual Warehouse M. Anvari Page 60

Virtual Warehouse M. Anvari Page 61

Virtual Warehouse A Virtual Data Warehouse approach is often chosen when there are infrequent demands for data and management wants to determine if/how users will use operational data. M. Anvari Page 62

Virtual Warehouse One of the weaknesses of a Virtual Data Warehouse approach is that user queries are made against operational DBs. One way to minimize this problem is to build a “Query Monitor” to check the performance characteristics of a query before executing it. M. Anvari Page 63

Distributed Data Warehouse M. Anvari Page 64

Distributed Data Warehouse A Distributed Data Warehouse is similar in most respects to a Central Data Warehouse, except that the data is distributed to separate mini-Data Warehouses (Data Marts ) on local or specialized servers M. Anvari Page 65

Information Access Tools l Desktop DBs l Spreadsheets l 4 GL/Desktop Query Tools l Decision Support Systems (DSS) l Multi-dimensional DBs (MDDs) l OLAP (On-line Analytical Processing l Executive Information Systems (EIS) l Data Visualization Tools l Data Mining Tools l Business Modeling and Simulation Tools M. Anvari Page 66

Data Warehousing Tools and Technology Desktop Data Bases: • Structured for Database Manipulation • Provides facility for selecting, and loading of Desktop DBs from Informational DBs • Provides ability to Create Highly “Personalized” Informational Systems Examples • Access • Paradox • d. Base/Fox. Pro/Clipper M. Anvari Page 67

Enterprise Network Computer Architecture Spreadsheets: • Structured to get any subset of Information • Ability to Interface with standard Spreadsheet tools ( Examples • Excel • 1 -2 -3 • Quatro Pro M. Anvari Page 68

Enterprise Network Computer Architecture Ad Hoc Query Systems: • Tailored for Flexible Reporting • Ability to do Sophisticated Analysis Functions • Aimed a a variety of users from casual to the power user Examples • Focus for Windows (IBI) • SAS • Business Objects • GQL (Anadyne) • Esperant (Software AG) • Forrest & Trees (Platinum) • Visualizer (IBM) • Impromptu (Cognos) • Beacon (Prodea) M. Anvari Page 69

Enterprise Network Computer Architecture Multi-dimensional Databases (MDDB) OLAP (On-line analytical processing): • Highly Structured Data • Tailored for Financial Modeling • Tailored for “Power Users” • Ability to do Sophisticated Financial “What-if” Analysis • Ability to “drill-down” from high-level to Detail Data Examples • Acumate (Kenan Tech. ) • Beacon (Prodea) • Cross. Target (Dimensional Insight) • e. SSbase (Arbor) • Oracle Express (Oracle) M. Anvari Page 70

Enterprise Network Computer Architecture Executive Information Systems (EIS): • Highly Structured Data • Tailored for Non-technical Users • Ability to “slice and dice” data • Ability to “drill-down” Examples • Commander OLAP Server • Pilot (Lightship) • VB • Powerbuilder M. Anvari Page 71

Enterprise Network Computer Architecture Data Visualization: • Automatic Categorization • Visualization of Multi-dimensional data • Automatic Analysis and/or Indexing Examples • Win. Viz (IBI) • db. Express (Computer Concepts) • Data Explorer (IBM) • ARC Info/ARC View • Strategic Mapping M. Anvari Page 72

Enterprise Network Computer Architecture Data Mining: • High Speed Analysis of Detail Data • Constructs Business Patterns • Provides Statistical Support Examples • IBM beta-test • Information Harvester • IDIS • d. b. Express • Data. Mind M. Anvari Page 73

Enterprise Network Computer Architecture Business Modeling and Simulation: • Business Feedback Model • Direct Manipulation • Business Gaming • Management/Operations Training Examples • Sim. Refinery • Sim. Telephone • i. Think • Microworlds M. Anvari Page 74

3. Meta-data Repository Layer Data Dictionary/ Repository • Meta-data Modeling • Meta-data Updating • Meta-data Examples o Platinum o Rochade o MSP o Data Atlas (IBM) o MS/TI M. Anvari Page 75

3. Process (Systems) Management Process Management • Scheduling • Execution • Subscription Examples o Data Harvester o Data Hub o Detect and Alert (Comshare) M. Anvari Page 76

3. Post-processing/Indexing Layer Post-processing/ Indexing Examples • Sybase IQ Accelerator • OMNIdex • Oracle 7. 3 • e. SSbase • IRI Express M. Anvari Page 77