Скачать презентацию The Big Deal About Big Data db 2 Скачать презентацию The Big Deal About Big Data db 2

c37e5509b92abc3d4373f8c4ca27c755.ppt

  • Количество слайдов: 43

The Big Deal About Big Data @db 2 Dean facebook. com/db 2 Dean www. The Big Deal About Big Data @db 2 Dean facebook. com/db 2 Dean www. db 2 Dean. com Dean Compher Data Management Technical Professional for UT, NV [email protected] ibm. com www. db 2 Dean. com 17 March 2018 Slides Created and Provided by: • Paul Zikopoulos • Tom Deustch © 2012 IBM Corporation

Why Big Data How We Got Here 17 March 2018 © 2012 IBM Corporation Why Big Data How We Got Here 17 March 2018 © 2012 IBM Corporation

3 3 © 2012 IBM Corporation …by the end of 2011, this was about 3 3 © 2012 IBM Corporation …by the end of 2011, this was about 30 billion and growing even faster In 2005 there were 1. 3 billion RFID tags in circulation…

An increasingly sensor-enabled and instrumented business environment generates HUGE volumes of data with MACHINE An increasingly sensor-enabled and instrumented business environment generates HUGE volumes of data with MACHINE SPEED characteristics… 4 1 BILLION lines of code EACH engine generating 10 TB every 30 minutes! © 2012 IBM Corporation

350 B Transactions/Year Meter Reads every 15 min. 120 M – meter reads/month 5 350 B Transactions/Year Meter Reads every 15 min. 120 M – meter reads/month 5 3. 65 B – meter reads/day © 2012 IBM Corporation

§ In August of 2010, Adam Savage, of “Myth Busters, ” took a photo § In August of 2010, Adam Savage, of “Myth Busters, ” took a photo of his vehicle using his smartphone. He then posted the photo to his Twitter account including the phrase “Off to work. ” § Since the photo was taken by his smartphone, the image contained metadata revealing the exact geographical location the photo was taken § By simply taking and posting a photo, Savage revealed the exact location of his home, the vehicle he drives, and the time he leaves for work 6 © 2012 IBM Corporation

The Social Layer in an Instrumented Interconnected World 30 billion RFID 12+ TBs tags The Social Layer in an Instrumented Interconnected World 30 billion RFID 12+ TBs tags today (1. 3 B in 2005) of tweet data every day camera phones world wide ? TBs of data every day 100 s of millions of GPS enabled devices sold annually 2+ billion 25+ TBs of log data every day 76 million smart meters in 2009… 200 M by 2014 7 4. 6 billion people on the Web by end 2011 © 2012 IBM Corporation

Twitter Tweets per Second Record Breakers of 2011 8 © 2012 IBM Corporation Twitter Tweets per Second Record Breakers of 2011 8 © 2012 IBM Corporation

Extract Intent, Life Events, Micro Segmentation Attributes Pauline Name, Birthday, Family Tom Sit Not Extract Intent, Life Events, Micro Segmentation Attributes Pauline Name, Birthday, Family Tom Sit Not Relevant - Noise Tina Mu Monetizable Intent Jo Jobs Not Relevant - Noise Location 9 Wishful Thinking Relocation SPAMbots Monetizable Intent © 2012 IBM Corporation

Big Data Includes Any of the following Characteristics Extracting insight from an immense volume, Big Data Includes Any of the following Characteristics Extracting insight from an immense volume, variety and velocity of data, in context, beyond what was previously possible Variety: Manage the complexity of data in many different structures, ranging from relational, to logs, to raw text Velocity: Streaming data and large volume data movement Volume: Scale from Terabytes to Petabytes (1 K TBs) to Zetabytes (1 B TBs) 10 © 2012 IBM Corporation

Bigger and Bigger Volumes of Data § Retailers collect click-stream data from Web site Bigger and Bigger Volumes of Data § Retailers collect click-stream data from Web site interactions and loyalty card data – This traditional POS information is used by retailer for shopping basket analysis, inventory replenishment, +++ – But data is being provided to suppliers for customer buying analysis § Healthcare has traditionally been dominated by paper-based systems, but this information is getting digitized § Science is increasingly dominated by big science initiatives – Large-scale experiments generate over 15 PB of data a year and can’t be stored within the data center; sent to laboratories § Financial services are seeing large and large volumes through smaller trading sizes, increased market volatility, and technological improvements in automated and algorithmic trading § Improved instrument and sensory technology – Large Synoptic Survey Telescope’s GPixel camera generates 6 PB+ of image data per year or consider Oil and Gas industry 11 © 2012 IBM Corporation

The Big Data Conundrum § The percentage of available data an enterprise can analyze The Big Data Conundrum § The percentage of available data an enterprise can analyze is decreasing proportionately to the available to it § Quite simply, this means as enterprises, we are getting “more naive” about our business over time § We don’t know what we could already know…. Data AVAILABLE to an organization Data an organization can PROCESS 12 © 2012 IBM Corporation

Why Not All of Big Data Before: Didn’t have the Tools? 13 © 2012 Why Not All of Big Data Before: Didn’t have the Tools? 13 © 2012 IBM Corporation

Applications for Big Data Analytics Smarter Healthcare Multi-channel sales Finance Log Analysis Homeland Security Applications for Big Data Analytics Smarter Healthcare Multi-channel sales Finance Log Analysis Homeland Security Traffic Control Telecom Search Quality Fraud and Risk Retail: Churn, NBO Manufacturing 14 Trading Analytics © 2012 IBM Corporation

Most Requested Uses of Big Data § Log Analytics & Storage § Smart Grid Most Requested Uses of Big Data § Log Analytics & Storage § Smart Grid / Smarter Utilities § RFID Tracking & Analytics § Fraud / Risk Management & Modeling § 360° View of the Customer § Warehouse Extension § Email / Call Center Transcript Analysis § Call Detail Record Analysis § +++ 15 15 © 2012 IBM Corporation

So What Is Hadoop? 16 © 2012 IBM Corporation So What Is Hadoop? 16 © 2012 IBM Corporation

Hadoop Background § Apache Hadoop is a software framework that supports dataintensive applications under Hadoop Background § Apache Hadoop is a software framework that supports dataintensive applications under a free license. It enables applications to work with thousands of nodes and petabytes of data. Hadoop was inspired by Google Map/Reduce and Google File System papers. § Hadoop is a top-level Apache project being built and used by a global community of contributors, using the Java programming language. Yahoo has been the largest contributor to the project, and uses Hadoop extensively across its businesses. § Hadoop is a paradigm that says that you send your application to the data rather than sending the data to the application 17 17 © 2012 IBM Corporation

What Hadoop Is Not § It is not a replacement for your Database & What Hadoop Is Not § It is not a replacement for your Database & Warehouse strategy – Customers need hybrid database/warehouse & hadoop models § It is not a replacement for your ETL strategy – Existing data flows aren’t typically changed, they are extended § It is not designed for real-time complex event processing like Streams – Customers are asking for Streams & Big. Insights integration 18 © 2012 IBM Corporation

So What Is Really New Here? § Cost effective / Linear Scalability. – Hadoop So What Is Really New Here? § Cost effective / Linear Scalability. – Hadoop brings massively parallel competing to commodity servers. You can start small and scales linearly as your work requires. – Storage and Modeling at Internet-scale rather than small sampling – Cost profile for super-computer level compute capabilities – Cost per TB of storage enables superset of information to be modeled § Mixing Structured and Unstructured data. – Hadoop is its schema-less so it doesn’t care about the form the data stored is in, and thus allows a super-set of information to be commonly stored. Further, Map. Reduce can be run effectively on any type of data and is really limited by the creatively of the developer. – Structure can be introduced at the Map. Reduce run time based on the keys and values defined in the Map. Reduce program. Developers can create jobs that against structured, semi-structured, and even unstructured data. § Inherently flexible of what is modeled/analytics run – Ability to change direction literally on a moment’s notice without any design or operational changes – Since hadoop is schema-less, and can introduce structure on the fly, the type of analytics and nature of the questions being asked can be changed as often as needed without upfront cost or latency 19 © 2012 IBM Corporation

Break It Down For Me Here… § Hadoop is a platform and framework, not Break It Down For Me Here… § Hadoop is a platform and framework, not a database – It uses both the CPU and disc of single commodity boxes, or node – Boxes can be combined into clusters – New nodes can be added as needed, and added without needing to change the; • Data formats • How data is loaded • How jobs are written • The applications on top 20 © 2012 IBM Corporation

So How Does It Do That? At its core, hadoop is made up of; So How Does It Do That? At its core, hadoop is made up of; § Map/Reduce – How hadoop understands and assigns work to the nodes (machines) § Hadoop Distributed File System = HDFS – Where hadoop stores data – A file system that’s runs across the nodes in a hadoop cluster – It links together the file systems on many local nodes to make them into one big file system 21 © 2012 IBM Corporation

What is HDFS § The HDFS file system stores data across multiple machines. § What is HDFS § The HDFS file system stores data across multiple machines. § HDFS assumes nodes will fail, so it achieves reliability by replicating data across multiple nodes – Default is 3 copies • Two on the same rack, and one on a different rack. § The filesystem is built from a cluster of data nodes, each of which serves up blocks of data over the network using a block protocol specific to HDFS. – They also serve the data over HTTP, allowing access to all content from a web browser or other client – Data nodes can talk to each other to rebalance data, to move copies around, and to keep the replication of data high. 22 © 2012 IBM Corporation

File System on my Laptop 23 © 2012 IBM Corporation File System on my Laptop 23 © 2012 IBM Corporation

HDFS File System Example 24 © 2012 IBM Corporation HDFS File System Example 24 © 2012 IBM Corporation

Map/Reduce Explained § Map/Reduce Explained § "Map" step: – The program is chopped up into many smaller subproblems. • A worker node processes some subset of the smaller problems under the global control of the Job. Tracker node and stores the result in the local file system where a reducer is able to access it. § "Reduce" step: – Aggregation • The reduce aggregates data from the map steps. There can be multiple reduce tasks to parallelize the aggregation, and these tasks are executed on the worker nodes under the control of the Job. Tracker. 25 25 © 2012 IBM Corporation

The Map. Reduce Programming Model § The Map. Reduce Programming Model § "Map" step: – Program split into pieces – Worker nodes process individual pieces in parallel (under global control of the Job Tracker node) – Each worker node stores its result in its local file system where a reducer is able to access it § "Reduce" step: – Data is aggregated (‘reduced” from the map steps) by worker nodes (under control of the Job Tracker) – Multiple reduce tasks can parallelize the aggregation 26 26 © 2012 IBM Corporation

Map/Reduce Job Example 27 © 2012 IBM Corporation Map/Reduce Job Example 27 © 2012 IBM Corporation

Map Shuffle Reduce § § § Murray 38 Salt Lake 39 Bluffdale 35 Sandy Map Shuffle Reduce § § § Murray 38 Salt Lake 39 Bluffdale 35 Sandy 32 Salt Lake 42 Murray 31 § § Murray 38 Bluffdale 35 Sandy 32 Salt Lake 42 § § Murray 38 Bluffdale 35 Bluffdale 37 Murray 30 § § § § Bluffdale 32 Sandy 40 Murray 27 Salt Lake 25 Bluffdale 37 Sandy 32 Salt Lake 23 Murray 30 § § Sandy 40 Salt Lake 25 Bluffdale 37 Murray 30 § § Sandy 40 Salt Lake 25 Sandy 32 Salt Lake 42 28 § Murray 38 § Bluffdale 37 § Sandy 40 § Salt Lake 42 © 2012 IBM Corporation

Map. Reduce In more Detail § Map-Reduce applications specify the input/output locations and supply Map. Reduce In more Detail § Map-Reduce applications specify the input/output locations and supply map and reduce functions via implementations of appropriate Hadoop interfaces, such as Mapper and Reducer. § These, and other job parameters, comprise the job configuration. The Hadoop job client then submits the job (jar/executable, etc. ) and configuration to the Job. Tracker § The Job. Tracker then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client. § The Map/Reduce framework operates exclusively on pairs — that is, the framework views the input to the job as a set of pairs and produces a set of pairs as the output of the job, conceivably of different types. § The vast majority of Map-Reduce applications executed on the Grid do not directly implement the low-level Map-Reduce interfaces; rather they are implemented in a higher-level language, such as Jaql, Pig or Big. Sheets 29 © 2012 IBM Corporation

Job. Tracker and Task. Trackers § Map/Reduce requests are handed to the Job Tracker Job. Tracker and Task. Trackers § Map/Reduce requests are handed to the Job Tracker which is a master controller for the map and reduce tasks. – Each worker node contains a Task Tracker process which manages work on the local node. – The Job Tracker pushes work out to the Task Trackers on available worker nodes, striving to keep the work as close to the data as possible – The Job Tracker knows which node contains the data, and which other machines are nearby – If the work cannot be hosted on the actual node where the data resides, priority is given to nodes in the same rack – This reduces network traffic on the main backbone network. If a Task Tracker fails or times out, that part of the job is rescheduled 30 30 © 2012 IBM Corporation

Skill Required How To Create Map/Reduce Jobs § Map/reduce development in Java – Hard, Skill Required How To Create Map/Reduce Jobs § Map/reduce development in Java – Hard, few resources that know this § Pig – Open source language / Apache sub-project – Becoming a “standard” § Hive – Open source language / Apache sub-project – Provides a SQL-like interface to hadoop § Jaql – IBM Research Invented – More powerful than Pig when dealing with loosely structure data – Visa has been a development partner § Big. Sheets – Big. Insights browser based application – Little development required – You’ll use this most often 31 © 2012 IBM Corporation

Taken Together - What Does This Result In? § Easy To Scale – Simply Taken Together - What Does This Result In? § Easy To Scale – Simply add machines as your data and jobs require § Fault Tolerant and Self-Healing – Hadoop runs on commodity hardware and provides fault tolerance through software. – Hardware losses are expecting and tolerated – When you lose a node the system just redirects work to another location of the data and nothing stops, nothing breaks, jobs, applications and users don’t even know. § Hadoop Is Data Agnostic – Hadoop can absorb any type of data, structured or not, from any number of sources. – Data from many sources can be joined and aggregated in arbitrary ways enabling deeper analyses than any one system can provide. – Hadoop results can be consumed by any system necessary if the output is structured appropriately § Hadoop Is Extremely Flexible – – 32 Start small, scale big You can turn nodes “off” and use for other needs if required (really) Throw any data, in any form or format, you want at it What you use it for can be changed on a whim © 2012 IBM Corporation

The IBM Big Data Platform 33 © 2012 IBM Corporation The IBM Big Data Platform 33 © 2012 IBM Corporation

Analytic Sandboxes – aka “Production” § Hadoop capabilities exposed to LOB with some notion Analytic Sandboxes – aka “Production” § Hadoop capabilities exposed to LOB with some notion of IT support § Not really production in an IBM sense § Really “just” ad-hoc made visible to more users in the organization § Formal declaration of direction as part of the architecture § “Use it, but don’t count on it” § Not built for secutity 34 © 2012 IBM Corporation

Production Usage with SLAs § SLA driven workloads – Guaranteed job completion – Job Production Usage with SLAs § SLA driven workloads – Guaranteed job completion – Job completion within operational windows § Data Security Requirements – – Problematic if it fails or looses data True DR becomes a requirements Data quality becomes an issue Secure Data Marts become a hard requirement § Integration With The Rest of the Enterprise – Workload integration becomes an issue § Efficiency Becomes A Hot Topic – Inefficient utilization on 20 machines isn’t an issue, on 500 or 1000+ it is § Relatively few are really here yet outside of Facebook, Yahoo, Linked. In, etc… § Few are thinking of this but it is inevitable 35 © 2012 IBM Corporation

IBM – Delivers a Platform Not a Product § Hardened Environment – – – IBM – Delivers a Platform Not a Product § Hardened Environment – – – Removes single points of failure Security All Components Tested Together Operational Processes Ready for Production § Mature / Pervasive usage § Deployed and Managed Like Other Mature Data Center Platforms § BIG INSIGHTS – Text Analytics, Data Mining, Streams, Others 36 © 2012 IBM Corporation

The IBM Big Data Platform Info. Sphere Big. Insights Hadoop-based low latency analytics for The IBM Big Data Platform Info. Sphere Big. Insights Hadoop-based low latency analytics for variety and volume Hadoop Information Integration Stream Computing Info. Sphere Information Server Info. Sphere Streams Low Latency Analytics for streaming data High volume data integration and transformation MPP Data Warehouse IBM Info. Sphere Warehouse IBM Netezza High Capacity Appliance Large volume structured data analytics Queryable Archive Structured Data 37 IBM Netezza 1000 BI+Ad Hoc Analytics on Structured Data IBM Smart Analytics System IBM Informix Timeseries Time-structured analytics Operational Analytics on Structured Data © 2012 IBM Corporation

What Does a Big Data Platform Do? Analyze a Variety of Information Novel analytics What Does a Big Data Platform Do? Analyze a Variety of Information Novel analytics on a broad set of mixed information that could not be analyzed before Analyze Information in Motion Streaming data analysis Large volume data bursts and ad-hoc analysis Analyze Extreme Volumes of Information Cost-efficiently process and analyze PBs of information Manage & analyze high volumes of structured, relational data Discover and Experiment Ad-hoc analytics, data discovery and experimentation Manage and Plan Enforce data structure, integrity and control to ensure consistency for repeatable queries 38 © 2012 IBM Corporation

Big Data Enriches the Information Management Ecosystem Active Archive Cost Optimization Master Data Enrichment Big Data Enriches the Information Management Ecosystem Active Archive Cost Optimization Master Data Enrichment via Life Events, Hobbies, Roles, +++ Establishing Information as a Service Audit Map. Reduce Jobs and tasks OLTP Optimization (SAP, checkout, +++) 39 Who Ran What, Where, and When? Managing a Governance Initiative © 2012 IBM Corporation

Get More Information… 17 March 2018 © 2012 IBM Corporation Get More Information… 17 March 2018 © 2012 IBM Corporation

www. bigdatauniversity. com 41 © 2012 IBM Corporation www. bigdatauniversity. com 41 © 2012 IBM Corporation

Get the Book 42 © 2012 IBM Corporation Get the Book 42 © 2012 IBM Corporation

43 © 2012 IBM Corporation 43 © 2012 IBM Corporation