Ocean Observatories Initiative OOI Cyberinfrastructure Release 1 Scope

Ocean Observatories Initiative OOI Cyberinfrastructure Release 1 Scope OOI CI - EOI User Review 16 June 2011 EOI User Review June 16 2011

OOI Integrated Observatory Requirements: Provide one integrated observatory interface to all users inside and outside the OOI. Enable the users to investigate observations, manage the observatory and its assets and collaborate with each other in teams. Engineering Drivers: A geographically distributed system of systems with observatories at multiple scales and operational authority. OOIwide need for data distribution, storage, processing and command control. EOI User Review June 16 2011

Virtual Observatories Observatory Interface Observatory Requirements: Enable communities of users to work with the instruments and observatory assets they need from across all OOI sites in a uniform observatory environment. Support multiple “virtual” observatories in parallel. Engineering Drivers: Multiple heterogeneous communities of users accessing shared observatory assets across the OOI. Enable access to observatory assets based on user authorization level and observatory policy. EOI User Review June 16 2011

OOI Deployment Topology Observatory Requirements Provide observatory assets to support measurements of scientific processes. Provide computational and network assets to enable realtime, secure access and processing for the observatory users Engineering Drivers: Proximity to observatory assets, geographic redundancy. Highbandwidth, low latency access to national and international network peering points. Access to commercial and academic compute clouds EOI User Review June 16 2011

National & International Observatory Integration NEPTUNE Canada Engineering Driver: The topology used by the OOI program is easily integrated into national and international cyber-networks. This will facilitate collaboration for developing the international ocean observing network NSF OOI Network NOAA N-Wave Network Global Lambda ntegrated Facility EOI User Review June 16 2011

Network Messaging Infrastructure • Based on the Layer 2 National Network Infrastructure • Provides direct application-level messaging connectivity between components (Capability Containers) within the Integrated Observatory Network, and at users’ sites. EOI User Review June 16 2011

What's Different about OOI? Integrated Observatory Network EOI User Review June 16 2011

Event Detection and Response Observatory Requirements: Support closed loop scientific activities. Enable interactive and event-driven control of the observatory assets based on interactive and real-time observations and analyses. Engineering Drivers: Workflows accessing real-time measurements, external data sources, real-time processing, QC and event detection, complex numerical models leading to control of mobile and stationary observatory assets. EOI User Review June 16 2011

Integrated Observatory Network: 4 Releases Integrated Observatory Network R 1 R 2 R 4 R 3 R 1: Data Distribution Network R 2: Managed Instrument R 3: On-Demand Measurement Network Processing R 4: Interactive Ocean Observatory EOI User Review June 16 2011

Release 1 of ION: Data Distribution Network • publish data to ION 'User-visible' Highlights • register data sources • find and browse data resources • examine data resources • initiate data notifications and subscriptions • download data from system • get live data streams • create user accounts • scale computing resources • perform diagnostics and monitoring • receive events without delay EOI User Review June 16 2011

Release 1 of ION: Data Distribution Network Technical Perspective • “Ingest” data products from external data sources and • • experimentally from sensors in canonical and raw formats Characterize data sources with their metadata attributes (format, structure) Distribute data via streaming and DAP servers to consumers such as data analysts, numerical modelers Provide a platform for instrument integration development, with instrument control and sensor data acquisition EOI User Review Provide a distributed service integration and execution June 16 2011

Release 1 of ION: Data Distribution Network Structural Highlights • Data distribution via publish/subscribe • messages received are immediately distributed • processes act on messages as soon as received (workflow!) • Scalability: manage growing amounts of work in a graceful way • new process host (virtual machines) can be started as needed • sets up for automated compute resource management • Fault tolerance: continue operation when part of system EOI User Review fails June 16 2011

Release 1 Components https: //confluence. oceanobservatories. org/display/syseng/CIAD+AV+Transition EOI User Review June 16 2011

CI Components Developed • • • Python Capability Container Java Capability Container Access Library Web UI Platform Exchange Messaging System Distributed State Infrastructure Resource Registry Framework Data Publish-Subscribe Framework Event Notification Framework Science Data Persistence and Transport Format Instrument Agent Framework Elastic Processing Unit Virtualized Cloud Management Tools https: //confluence. oceanobservatories. org/display/syseng/CIAD+AV+Transition EOI User Review June 16 2011

Use Cases for Release 1 Basic Connections UC. R 1. 01 Hello User UC. R 1. 02 Hello Instrument UC. R 1. 03 Hello Data Source UC. R 1. 18 Command An Instrument UC. R 1. 20 Command A Resource Data UC. R 1. 14 UC. R 1. 04 Ingest and Describe Data UC. R 1. 15 Derive Data Product UC. R 1. 21 UC. R 1. 16 Externally (Merge Data) UC. R 1. 17 UC. R 1. 09 Discover Resource Services & Operations Use Service Anywhere Put Services Anywhere Scale the Processing Replicate Service UC. R 1. 05 Synchronize State Data UC. R 1. 25 Assure Reliability UC. R 1. 06 Distribute Data Product UC. R 1. 28 Operate System UC. R 1. 07 Subscribe To Data UC. R 1. 29 Monitor System UC. R 1. 08 Persist Streamed Data UC. R 1. 30 Troubleshoot System EOI User Review June 16 2011

EOI User Review June 16 2011

"Hello User" Use Case 1. User registers with the system by providing an (external) identity, creating an internal identity for user. • Internally, user has an internal identity and a user "profile"; which contains 'life cycle state' of the user and other user attributes such as contact information and preferences. • User authenticates to external identity provider, in order to assert external identity • Background: The In. Common federation facilitates the exchange of knowledge that enables the assertion of identities across organizations within this federation. The NSF CI-Logon project is developing cyberinfrastructure software to apply In. Common processes within CI systems, such as OOI. A user, for instance from UCSD, will be able to use her organization credentials to register with OOI, and subsequently use OOI credentials. • Each user identity is tracked in the user identity repository. 2. User logs in to the system using identity credential (e. g. , login ID and password). 3. User accesses the system's web user interface, which calls system services. • Features: Status, Register Resources, Search (User, Instrument, Data, Process), Instrument Access 4. User is assigned their EOI User Review June 16 role default 2011 in system

Implications of OOI Approach • We have user identities in the system, enabling security • We keep user attributes of interest to us (e. g. , contact info) • We assign attributes as needed (e. g. , user roles) • ION doesn't perform authentication itself, so … • Users don't have to remember a custom OOI identity • ION doesn't have to maintain user credentials internally • passwords and associated security issues • support for all the associated human issues • software we don't have to write, or maintain • This solves a big scaling issue EOI User Review June 16 2011

Select Identity Provider and Log On Cornell University Duke University Georgetown University Google EOI User Review June 16 2011

Identity Provider Logs You In We get a certificate saying who (Google says) you are. EOI User Review June 16 2011

EOI User Review June 16 2011

Data Distribution Use Case 1. Data is accepted at a specific Acquisition Point Cyber. Po. P or any specific capability container. • External data will be accepted via an Instrument Agent or other adapter process, and go through an acquisition process before ingested as complete data set or increment for a data set. • Information does not have to be ingested (i. e. made available in the resource registry, normalized) to be distributed via the Data Distribution Network 2. Once minimally described, data is sent to the Exchange by the Data Producer. • Ideally data should be fully described before routing, not just minimally described. • Further description of the data may already be present in the resource registry of data sets. 3. The Exchange routes data messages to all subscribers to a data stream or filter that match certain criteria in the description of the data message • Matching certain criteria is also fundamental to several other use cases, including explicitly in Version a Resource. 4. Subscribers equally see data, whether internal (local or remote) or external users. • A data store (or several different stores) is a subscriber to most forms of ingested science data, in no way different to any other subscriber. In case there is no data store subscriber, data will be distributed across the Data Distribution Network without any form of persistence; subscribers are solely responsible for EOI User Review maintaining the information they need. June 16 2011

Subscribe to Data Use Case 1. Data Consumer identifies material of interest. 2. Variant A: Data Consumer selects existing data stream that provides the material of interest. 3. Variant B: Data Consumer selects historic data set resource … 4. Data Consumer registers (through API or web form) with Subscription service to be delivered material of interest, per a defined set of criteria. 5. Data Consumer specifies subscription delivery modality with Subscription service. 6. Registration is acknowledged via message returned to Data Consumer. 7. Registration is verified through inspection of the Inventory and Data Stream Registry. 8. Data messages meeting registered criteria arrive within Exchange, where they are routed to subscribed Data Consumer. 9. Event notification messages generated from event producing processes (e. g. Instrument Agents, Data Source adapter, event aggregator), arrive within the Exchange, where they are routed to registered Data Consumer. EOI User Review June 16 2011

Implications of "Data Distribution" • User gets data when we get data (no waiting!) • Users or modules can subscribe to selected data • Processing routines kick off when data arrives … workflows! • Fundamental artifacts are data streams (not static sets) • ION internalizes the data in its constituent elements • Allows fully using the data's natural structure • Can present the data according to users' needs • All kinds of data sources: models, sensors, observatories, people, … • ION can keep a 'reference copy' for fast, reliable access EOI User Review June 16 2011 • The data, and products, can be well curated (good metadata)

CI System Design Engineering Milestones • Passed Life Cycle Objectives (LCO) Review for Release-1 (Mar 2010) • Approval of Release-1 system architecture baseline and use cases • Demonstration of critical technologies for risk mitigation • Passed Life Cycle Architecture (LCA) Review for Release-1 (Aug 2010) • Approval of refined Release-1 use cases and system architecture for Construction • Demonstration of an end-to-end running system and a scalable architecture • Passed 2 subsystem LCA reviews: Instrument Platform Agent Architecture, External Observatory Integration (December 2010) • Passed Initial Operational Capability (IOC) Review (May 2011) • Up-to-date approved architecture and design • Added Cyber. Po. P and network design; user experience and user interface designs EOI User Review June 16 2011

CI Development Progress by Subsystem ID Subsystem # R 1 L 4 Req’s Satisfied # R 1 Use Cases Satisfied % Design complete SLOC # Tasks complete SA Sensing & Acquisition 3 (7%) 3 (75%) 39% 1, 198 30 AS Analysis & Synthesis N/A N/A 0 0 PP Planning & Prosecution N/A N/A 0 0 DM Data Management 41 (59%) 5 (56%) 95% 23, 139 110 COI Common Operating Infrastructure 31 (26%) 4 (36%) 95% 34, 467 155 CEI Common Execution Infrastructure 14 (56%) 8 (100%) 77% 35, 599 109 IPA Instrument and Platform Agents 14 (41%) 1 (100%) 100% 11, 587 52 EOI IOOS Integration Package 1 15 (100%) 100% 7, 793 24 N/A 0 (0%) 80% 24, 701 67 ITV Integration, Test and Verification N/A 100% 6, 984 N/A TCN Terrestrial Cyber. Po. Ps and Network N/A 100% N/A UX User Experience EOI User Review June 16 2011 41

EOI Presentation EOI User Review June 16 2011