Скачать презентацию SPRUCE d up LEAD An Urgent Computing Scenario Suresh Скачать презентацию SPRUCE d up LEAD An Urgent Computing Scenario Suresh

b05057561dfaa7cc0cee0411a769665c.ppt

  • Количество слайдов: 32

SPRUCE’d up LEAD An Urgent Computing Scenario Suresh Marru, Dennis Gannon, Beth Plale School SPRUCE’d up LEAD An Urgent Computing Scenario Suresh Marru, Dennis Gannon, Beth Plale School of Informatics, Indiana University Suman Nadella Argonne National Laboratory Indiana University School of Informatics

Tornados and Hurricanes • The loss of life and property due to tornados and Tornados and Hurricanes • The loss of life and property due to tornados and hurricanes each year is large and growing. • We seem to be entering a period where the situation may becoming worse. • We are also on a threshold of far more advanced and accurate prediction capabilities. Indiana University School of Informatics

The LEAD Project Indiana University School of Informatics The LEAD Project Indiana University School of Informatics

Traditional Methodology STATIC OBSERVATIONS Analysis/Assimilation Prediction/Detection Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Traditional Methodology STATIC OBSERVATIONS Analysis/Assimilation Prediction/Detection Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields PCs to Teraflop Systems Product Generation, Display, Dissemination The Process is Entirely Serial and Static (Pre-Scheduled): No Response to the Weather! End Users Indiana University School of Informatics NWS Private Companies Students

Am Major Paradigm Shift: CASA NETRAD adaptive Doppler Radars. Indiana University School of Informatics Am Major Paradigm Shift: CASA NETRAD adaptive Doppler Radars. Indiana University School of Informatics

The LEAD Vision: Adaptive Cyberinfrastructure Analysis/Assimilation Prediction/Detection Quality Control Retrieval of Unobserved Quantities Creation The LEAD Vision: Adaptive Cyberinfrastructure Analysis/Assimilation Prediction/Detection Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields DYNAMIC OBSERVATIONS PCs to Teraflop Systems Product Generation, Display, Dissemination Models and Algorithms Driving Sensors The CS challenge: Build cyberinfrastructure services that provide adaptability, scalability, availability, useability, and real -time response. Indiana University School of Informatics End Users NWS Private Companies Students

Dynamic Adaptation in LEAD: Overview • Respond to weather: Improvement over the current static Dynamic Adaptation in LEAD: Overview • Respond to weather: Improvement over the current static forecast execution model. • Identify prescribed features in real-time within radar reflectivity or radial velocity by operating on streaming NEXRAD Level II observations, and/or ADAS analyses. • Upon locating a region in which the criteria have been met, a single grid WRF forecast is automatically launched in an on-demand fashion on the Tera. Grid.

A Scientific Quest • An Example Exploration Scenario – A huge and unexpected storm A Scientific Quest • An Example Exploration Scenario – A huge and unexpected storm develops over the gulf of Mexico and moves ashore in Texas. As it moves northeast it spawns a series of unusual tornados along the eastern seaboard. • Unusual because the standard models did not predict them. What is going • An Atmospheric Scientist, Kelvin, wants to understand what is going wrong. Indiana University School of Informatics Wrong?

Need for Urgent Computing • For LEAD infrastructure to dynamically adapt to weather, computing Need for Urgent Computing • For LEAD infrastructure to dynamically adapt to weather, computing power is needed in short notice. • SPRUCE provides mechanism for obtaining resources when needed. Indiana University School of Informatics

How can we get cycles? • Build supercomputers for the application – Pros: Resource How can we get cycles? • Build supercomputers for the application – Pros: Resource is ALWAYS available – Cons: Incredibly costly (99% idle) – Example: Coast Guard rescue boats • Share public infrastructure – Pros: low cost – Cons: Requires complex system for authorization, resource management, and control – Examples: school buses for evacuation, cruise ships for temporary housing Urgent Computing -

Event SPRUCE Architecture Overview (1/2) Right-of-Way Tokens 2 Automated Trigger 1 First Responder SPRUCE Event SPRUCE Architecture Overview (1/2) Right-of-Way Tokens 2 Automated Trigger 1 First Responder SPRUCE Gateway / Web Services Right-of-Way Token Human Trigger Right-of-Way Token Urgent Computing -

SPRUCE Architecture Overview (2/2) Submitting Urgent Jobs User Team Authentication 4 Urgent Computing Job SPRUCE Architecture Overview (2/2) Submitting Urgent Jobs User Team Authentication 4 Urgent Computing Job Submission Conventional Job Submission Parameters ! Priority Job Queue Choose a Resource SPRUCE Job Manager 3 Local Site Policies Urgent Computing Parameters 5 Supercomputer Resource Urgent Computing -

The Search for Data • Kelvin first needs to see what data is available The Search for Data • Kelvin first needs to see what data is available – He goes to his weather gateway portal • A web portal to his data and science resources stored on the Grid. – He starts a search • “Show me pressure gradients for the gulf in the last 12 hours. ” – This gives a good start. Now he needs a more detailed look at the radar/forecast record for the north east. • He uses a different tool to graphically specify the details Indiana University School of Informatics

Data Search • Select a region and a time range and desired attributes Indiana Data Search • Select a region and a time range and desired attributes Indiana University School of Informatics

Try an experiment • Kelvin think “Hmmm. If the physics in the standard WRF Try an experiment • Kelvin think “Hmmm. If the physics in the standard WRF model is bad, I am going to try my new version with better physics. • To do so he must – add his version of the WRF application to the Grid – Reconfigure the workflow to use this version. Indiana University School of Informatics Let’s try my Secret new WRF version!

Kelvin uses the workflow composer to add his new service Kelvin’s wrf Indiana University Kelvin uses the workflow composer to add his new service Kelvin’s wrf Indiana University School of Informatics

The Experiment Builder • A Portal “wizard” that leads the user through the set-up The Experiment Builder • A Portal “wizard” that leads the user through the set-up of a workflow • Asks the user: – “Which workflow do you want to run? ” • Once this is know, it can prompt the user for the required input data sources • Then it “launches” the workflow. Indiana University School of Informatics

Parameter Selection Indiana University School of Informatics Parameter Selection Indiana University School of Informatics

Selecting the forecast region Indiana University School of Informatics Selecting the forecast region Indiana University School of Informatics

Indiana University School of Informatics Indiana University School of Informatics

The Tera. Grid • The US National Supercomputer Grid – Cyber. Infrastructure composed of The Tera. Grid • The US National Supercomputer Grid – Cyber. Infrastructure composed of a set of resources (compute and data) that provide common services for • Wide area data management (gridftp, gpfs, staged disk to tape. ) • Single sign-on user authentication (globus toolkit) • Distributed Job scheduling and management. (in the works. ) • Collectively – 1 Petaflop – 20 Petabytes • Soon to triple. • Will add a new petaflop machine each year. Indiana University School of Informatics

Final Results are automatically logged in the User’s metadata Catalog • • Cyber. Infrastructure Final Results are automatically logged in the User’s metadata Catalog • • Cyber. Infrastructure extends user’s desktop to incorporate vast data analysis space. As users go about doing scientific experiments, the CI manages back-end storage and compute resources. – Portal provides ways to explore this data and search and discover it. • Metadata about experiments is largely automatically generated, and highly searchable. – Describes data object (the file) in application-rich terms, and provides URI to data service that can resolve an abstract unique identifier to real, on-line data “file”. Indiana University School of Informatics

The Realization in Software Workflow graph Application services Compute Engine User Portal Workflow Engine The Realization in Software Workflow graph Application services Compute Engine User Portal Workflow Engine Fault Tolerance & scheduler Event Notification Bus Portal server My. LEAD Agent service Data Catalog service My. LEAD User Metadata catalog Data Management Service Providence Collection service Indiana University School of Informatics Data Storage

Experience so far • First release to support “Wx. Challenge: the new collegiate weather Experience so far • First release to support “Wx. Challenge: the new collegiate weather forecast challenge” – The goal: “forecast the maximum and minimum temperatures, precipitation, and maximum sustained wind speeds for select U. S. cities. – to provide students with an opportunity to compete against their peers and faculty meteorologists at 64 institutions for honors as the top weather forecaster in the nation. ” – 79 “users” ran 1, 232 forecast workflows generating 2. 6 TBybes of data. • Over 160 processors were reserved on Tungsten from 10 am to 8 pm EDT(EST), five days each week • National Spring Forecast – First use of user initiated 2 Km forecasts as part of that program. Generated serious interest from National Severe Storm Center. • Integration with CASA project scheduled for final year of LEAD ITR. Indiana University School of Informatics

Raising the Level of Discovery • Programming Knowledge Discovery – How can a user Raising the Level of Discovery • Programming Knowledge Discovery – How can a user pose data queries and rules that allow autonomous agents to monitor data streams and dynamic archives looking for patterns that trigger complex actions? • Watch the weather over Chicago and launch a storm forecast when a supercell is detected? • Mine all genome databases and search for strong genetic matches to a newly decoded genome. For those with highest scores, do an analysis of genetic annotation similarities. • Take the sequence of data analysis steps I just applied to this data and build a workflow and apply it to all similar data. Report results that match this pattern … Indiana University School of Informatics

Test the new idea in the “field” • Kelvin is happy with the new Test the new idea in the “field” • Kelvin is happy with the new algorithm • He decides to see if it works on a new storm… – So must he wait? – No. Ask the system to look for storm conditions that fit and then try the forecast simulation out. Indiana University School of Informatics

The Future Programming Model: Adaptive Queries LEAD requires ability to construct workflows that are The Future Programming Model: Adaptive Queries LEAD requires ability to construct workflows that are • Data Driven – Weather data streams define nature of computation • Persistent and Agile – Data mining of data stream, detects “interesting” feature, event triggers workflow scenario that has been waiting for months. • Adaptive – In response to weather: weather changes. – Nature of workflow may have to change on-the-fly. – Resource and requirements change. Indiana University School of Informatics

Conclusions • The Gateway concept is an ideal way to engage students, teachers, and Conclusions • The Gateway concept is an ideal way to engage students, teachers, and researchers in the use of advanced scientific applications. • For LEAD, the National Forecast Challenge. – Student teams can use lead to do detailed forecasts of their own without having to deal with acquiring access to massive computing resources. • For researchers, it enables a new modality of collaboration. Indiana University School of Informatics

Challenges • Defining a common workflow and data provenance model – 20 different workflow Challenges • Defining a common workflow and data provenance model – 20 different workflow systems – 10 different grid data provenance systems • Other Challenges – Security (specifically authorization and privacy) is still a major problem. – Propagation of software stack updates in heterogeneous Grids. Indiana University School of Informatics