Ocean Observatories Initiative OOI Cyberinfrastructure Architecture Overview Michael

Ocean Observatories Initiative OOI Cyberinfrastructure Architecture Overview Michael Meisinger Life Cycle Architecture Review La Jolla, CA OOI CI LCA REVIEW August 2010

Goals for this Presentation • To set the stage… • • • What is … the OOI Integrated Observatory? What is … the CI (Cyberinfrastructure)? What is … Release 1? What is … the role of the architecture in Construction? Where … can I find information about the CI design? • What … were the main accomplishments in Elaboration? • What … are the core implementation choices? • … for detailed subsystem presentations OOI CI LCA REVIEW August 2010 2

Overview • Architecture in the System Engineering Process • Architectural Overview • The OOI Integrated Observatory • CI Capabilities in Release 1 • Subsystems and Interfaces • Key Concepts • Capability Integration Strategy • Deployment: Plan and Strategy • Implementation of the Architecture • Status and Central Technology Choices OOI CI LCA REVIEW August 2010 3

Role of the Architecture • Blueprint for software construction • Defines long term intent • Defines the core structure of the system (interfaces, services, startup, data flow) and fundamental principles • Starting point and daily reference for developers during task definition each iteration • Place to document consolidated understanding and design refinement • Supporting OOI-wide SE processes • Requirements management • Interface management • Configuration control • Integration, test, validation OOI CI LCA REVIEW August 2010 4

Architecture Products • Architecture Documentation • Story, background, context (Living, consistent documentation) • Located on Confluence, exported to PDF • Architecture Model • Drawings, Definitions, Tracing • Authored in Enterprise Architect, clickable export • Searchable PDF document of all drawings • Technology List • Candidate technologies for integration, interfacing, reference • Living spreadsheet and reference catalog OOI CI LCA REVIEW August 2010 5

Architecture Documentation on Confluence OOI CI LCA REVIEW August 2010 6

Enterprise Architect: Drawings Used to: • Define • Draw • Connect • Identify • Check • Document • Browse • Export Examples: • Drawing repository • Requirements Tracing • Interface identification HTML Export: http: //ooici. net/eaexport/ Drawing Export PDF: Part of the materials OOI CI LCA REVIEW August 2010 7

OOI Ocean Observatory (full scope) OOI CI LCA REVIEW August 2010 8

OOI Integrated Observatory pe co 1 f s se o ut elea O r or f pe co 1 f s se o ut elea O r or f OOI CI LCA REVIEW August 2010 9

Release 1: Primary Target Audience External Data Sources (Observatories, such as IOOS with GTS/IDD, individual data producers) Selected groups of: Data Distribution (Publish – Subscribe) in OOI Common Data Model OOI CI LCA REVIEW August 2010 • Data Analysts • Numerical Modelers 10

Scope R 1: From Product Definition • Release 1: Data Distribution Network (May 2011) • “Ingest” data products from external data sources and • • • experimentally from sensors Characterize data sources with their metadata attributes (format, structure) Transform data into internal OOI data formats, for stream-based distribution and persistence Distribute data via streaming and DAP servers to consumers such as data analysts, numerical modelers Provide a platform for instrument integration development, w/ control and sensor data acquisition Provide a distributed service integration and execution platform in more than one programming languge OOI CI LCA REVIEW August 2010 11

System Structure IOOS Application Services Infrastructure Services 2650 -00007 OV 2 CI OOI CI LCA REVIEW August 2010 12

Core System Structure: Subsystems Sensing & Acquisition Data Management Common Operating Infrastructure Common Execution Infrastructure 2650 -00008 August 2010 OV 2 CI OOI CI LCA REVIEW 13

Core System Structure: Service Layers Sensing & Acquisition A&S P&P Data Management Common Operating Infrastructure Common Execution Infrastructure OOI CI LCA REVIEW August 2010 Application Level Services Data Management Infrastructure Services Core Infrastructure Services 14

Core System Structure: Service Layers Instrument Agents User Interfaces External Observatory Integration Application Level Services Sensing & Acquisition Data Management Common Operating Infrastructure Data Distribution and Preservation Infrastructure Common Execution Infrastructure OOI CI LCA REVIEW August 2010 Externalization and Integration Core Infrastructure Services 15

Core System Structure: Subsystems Instrument Agents User Interfaces External Observatory Integration Application Level Services Sensing & Acquisition Data Management Common Operating Infrastructure Externalization and Integration Data Distribution and Preservation Infrastructure Common Execution Infrastructure Core Infrastructure Services OOI CI LCA REVIEW August 2010 16

Deployment: The Capability Container • Ingredients: • Capability: services and processes (agent, adapter 0 • Plumbing: Lower level infrastructure: Messaging, security • Container: drop it all in as needed • Packaging: label it, tape it shut… done! OOI CI LCA REVIEW August 2010 17

Schematic: CI Interfaces and Subsystems OOI CI LCA REVIEW August 2010 18

Integration Strategy The Common Operating Infrastructure (COI) is the integration & communication environment for all the other subsystem services and agent/adapter processes OOI CI LCA REVIEW August 2010 19

Integration Strategy • Capability Container • Secure reliable messaging via the “Exchange” • Service-oriented architecture • Non-central governance • COI Infrastructure Services OOI CI LCA REVIEW August 2010 20

Exchange: Secure Reliable Messaging Capability (Service) Agent (Process) OOI CI LCA REVIEW August 2010 Agent (Process) UI Client (Process) 21

Deployment Plan and Strategy • Deployment of packaged capability containers • in OOI clusters and in the Amazon cloud • by leveraging virtualization • through the CEI subsystem infrastructure • Cyber. Po. P (Cyberinfrastructure Point of Presence): physical deployment of one or multiple purpose –built capability containers • Network connectivity • Storage and computation (redundant) OOI CI LCA REVIEW August 2010 22

Network Deployment OOI CI LCA REVIEW August 2010 23

Goals of Spiral Phases in Development • Inception • Define system objectives with stakeholders • Perform isolated prototyping in high risk areas • Get the development team trained in technologies and dependencies • Define the architecture and integration plan • Elaboration • Build out the core interfaces (messages) and architectural components (services) • Integrate services by exercising end-to-end scenarios • Refine and consolidate the architecture and design specs • Define a construction plan based on measured team performance • Construction • Refactor core interfaces and components based on lessons learned • Implement services and components according to construction plan • Successively add to the test suite to prove satisfaction of release use cases OOI CI LCA REVIEW August 2010 24

Progress in Elaboration • Refined and integrated the architecture with end-to-end scenarios. • Key achievement: Ideas hold and are feasible! • See demos • One primary code base “LCAarch” across all subsystems • Presentation Platform: Web UI • Interoperability with Java platform • Got development team entrained in the CI architectural vision and fundamental principles OOI CI LCA REVIEW August 2010 25

LCAarch: Architectural Integration Code Base • Primary technologies (more later): • Python 2. 5 programming language • AMQP messaging with Rabbit. MQ • Cent-OS Linux Operating System • Amazon EC 2 target execution environment • Development progress to date • More than 25 K lines of code • ~15 Developers across 6 teams • Initial integration across all subsystems • All core services defined; end-to-end scenarios exercised OOI CI LCA REVIEW August 2010 26

Proof of the Operational Architecture • Iterative, Test-Driven Development Approach • Functional testing • Coverage of central use cases of the Product Description • Comprehensive unit test suite: 340+ tests • Automatic test every 30 min • Deployment testing • Automatic test in EC 2 target deployment environment • Test in embedded platform environment: TS-7370, OK • Performance testing • Scalability test: successful scaling to 200+ server nodes OOI CI LCA REVIEW August 2010 27

LCAarch code base: Fish. Eye view OOI CI LCA REVIEW August 2010 28

Source Code Documentation Generation: Overview Diagram OOI CI LCA REVIEW August 2010 29

Enterprise Architect Roundtrip Engineering: Code View OOI CI LCA REVIEW August 2010 30

Further Presentations • Subsystems [w/ Overview, Progress, Technologies, Demos] • Common Operating Infrastructure • Common Execution Infrastructure • Data Management (Day 2) • Sensing and Acquisition • Implementation Projects • Instrument and Platform Agents • External Observatory Integration • Terrestrial Cyber. Pops and Network • Engineering Summary • Architectural Assessment OOI CI LCA REVIEW August 2010 31

Use Case by Subsystem Matrix Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 Name Hello User Hello Instrument Hello Data Source Ingest and Describe Data Synchronize State Data Distribute Data Product Subscribe to Data Persist Streamed Data Discover Resource Define Interaction Define New Service Annotate Data Transform Data Use Service Anywhere Put Services Anywhere Scale the Processing Replicate Service Command An Instrument Direct Instrument Access Command A Resource Derive Data Product Externally Present as Catalog Put to External Repository Version A Resource Assure Reliability Virtualize Everything Configure Access Page Operate System Monitor System Troubleshoot System Assess Access Policy Conduct Negotiation Enroll in an Org Share an Org Resource Share Affiliated Orgs' Resources Create an Org Description User gets an ID and logs in Instrument gets an ID and is hooked up Data source is registered and connected Externally provided data read and distributed Synchronize state in distributed data store Data made available to many consumers User finds data, asks for update notifications Store streamed data for a given time User searches for resources meeting criteria Describe pattern of interaction between actors Add new service to system capabilities Information or description is added to resource Data process produces new data from old Messages go to services wherever they are Allocate services where need is greatest Increase processing quickly to meet demand Configure service once, deploy many times Send typical commands to specific instrument Directly communicate with instrument Send typical commands to specific resource Provide data unified from many sources Present organized set of resources externally. Send data to special-purpose repository Resource is supplanted by changed version Computer fails, messages resent, work resumes Virtual processes embody all system services Configure user’s web page that accesses ION Configure system and respond to requests Anticipate issues using monitoring tools Diagnose issues using logs, feeds, tools Define access policy for a given resource Negotiate agreement (or not) between agents Enter as a member into an Organization (Org) Colloborate with Org member offering a resource Orgs form a contract to allow resource sharing Create an Organization (Org) with defined characteristics OOI CI LCA REVIEW August 2010 COI R R S S CEI U S R R S S S S R R R DM R U R R R S&A R R S U EOI U S U U IPAA S S UX S S S OPS S U U S R R R S S S R R S R U S S S R R R S U U U S S R S S S R 32

Thanks ! Questions ? OOI CI LCA REVIEW August 2010 33

OOI Domains of Authority (full scope) OOI CI LCA REVIEW August 2010 34

Scope of Future Releases • Release 2: Managed Instrument Network (2012) • Fully managed instrument activation and control; ready for deployment on OOI moorings and cable infrastructure • Qualified data products based on automated data processing • Release 3: On-Demand Measurement Processing (2013) • Observatory management and resource scheduling • On demand event driven measurements • Data processing workflows • Release 4: Integrated Modeling Network (2014) • Interactive analysis and synthesis • Numerical model integration pe co 1 s e of as ut ele O r or f • Release 5: Interactive Ocean Observatory (2015) • Full control of data, processes, models, instruments • Closed loop, adaptive sensing and platform control OOI CI LCA REVIEW August 2010 35

Preparing Services for Deployment Application Point Ingest Point Acquisition Point Access Portal Storage Point Instrument Point OOI CI LCA REVIEW August 2010 36

Deployment Scenario pe co 1 f s se t o lea u e O r or f OOI CI LCA REVIEW August 2010 37

External Interfaces OOI CI LCA REVIEW August 2010 1604 -00001 SV 1 OOI 38

pe co 1 f s se t o lea Ou re or f User CI External Interfaces Instance Cloud Execution Point Terrestrial Cyber. Po. P Marine Management CI Station pe o sc e 1 of as ut ele O rr fo CI Instance on platform pe co 1 f s se t o lea Ou re or f OOI CI LCA REVIEW August 2010 39

System Deployments 2660 -00014 SV 1 CI OOI CI LCA REVIEW August 2010 40

Release Schedule 6 OOI CI LCA REVIEW August 2010 41

Network Architecture OOI CI LCA REVIEW August 2010 42