Скачать презентацию Euclid Consortium OU SDC Proposed Development Strategy Maurice Скачать презентацию Euclid Consortium OU SDC Proposed Development Strategy Maurice

4965222f186f04bdcf7b82b9a4dfac8c.ppt

  • Количество слайдов: 23

Euclid Consortium OU /SDC Proposed Development Strategy Maurice Poncet (CNES, SDC-FR Lead) Andrea Zacchei Euclid Consortium OU /SDC Proposed Development Strategy Maurice Poncet (CNES, SDC-FR Lead) Andrea Zacchei (Inaf – SDC-IT Lead) Keith Noddle (If. A – SDC-GB Lead) OU-LE 3 KOM Meeting, London March 22 -23, 2012 1

Euclid – Ground Segment Concepts Euclid Consortium The recurring acronyms of the ground segment: Euclid – Ground Segment Concepts Euclid Consortium The recurring acronyms of the ground segment: OU-LE 3 KOM Meeting, London March 22 -23, 2012 2

Euclid – Ground Segment Concepts Euclid Consortium OUs (Organization Unit) : OU Responsible for Euclid – Ground Segment Concepts Euclid Consortium OUs (Organization Unit) : OU Responsible for the definition and prototyping of a given pipeline (i. e transform scientific req. from SWG into algorithms or mathematical methods) Responsible for the validation of a given pipeline (i. e providing baseline test data) SDC DEV SDCs (Science Data Centers) : Responsible for the S/W development of pipelines : SDC Dev Responsible for the H/W processing infrastructure : SDC Prod SDC PROD Responsible for the pipeline processing operations : SDC Prod OU-LE 3 KOM Meeting, London March 22 -23, 2012 3

SDCs - Development Strategy • Euclid Consortium SDC-xx DEV Team is a “delocalized” team SDCs - Development Strategy • Euclid Consortium SDC-xx DEV Team is a “delocalized” team – Distributed over several labs – Integration of other countries contributions • This team is lead by only one responsible which is at SDC-xx Dev level • SDC-xx DEV Team works through a collaborative web based development infrastructure • A given SDC-xx DEV Team may contribute to SDC of other countries (under their responsibility) • There is one dev team per pipeline • SOC should be involved as any other SDC OU-LE 3 KOM Meeting, London March 22 -23, 2012 4

SDCs – Implementation Involvement Euclid Consortium • L : Lead • C : Contribution SDCs – Implementation Involvement Euclid Consortium • L : Lead • C : Contribution (as corresp. OU Co lead) • c : other contribution OU-LE 3 KOM Meeting, London March 22 -23, 2012 5

Euclid Consortium SDCs – SDC Generic WPs WP-4 -4 -nn SDC-<NAT> WP-4 -4 -nn-1000 Euclid Consortium SDCs – SDC Generic WPs WP-4 -4 -nn SDC- WP-4 -4 -nn-1000 SDC- Management WP-4 -4 -nn-2000 SDC--DEV WP-4 -4 -nn-3000 SDC--PROD WP-4 -4 -nn-2100 SDC--DEV Infra. WP-4 -4 -nn-3100 SDC--PROD Infra. WP-4 -4 -nn-2200 SDC--DEV Implementation WP-4 -4 -nn-3200 SDC--PROD Operation WP-4 -4 -nn-22 sb pipeline Implementation Feel free to propose any other set up. WP-4 -4 -nn-22 sb-st pipeline Implementation subtask WP-4 -4 -nn-32 sb pipeline operation WP-4 -4 -nn-32 sb-st pipeline operation subtask WP-4 -4 -xx-2300 SDC--DEV Integration WP-4 -4 -xx-23 sb pipeline Integration WP-4 -4 -nn-23 sb-st pipeline Integration subtask OU-LE 3 KOM Meeting, London March 22 -23, 2012 6

Euclid Consortium SDC-DEV – Dev Environment SDC-DEV Local Infra Dev Dev Compilation Unit Test Euclid Consortium SDC-DEV – Dev Environment SDC-DEV Local Infra Dev Dev Compilation Unit Test Unit Tests S/W int tests SDC-PROD e – Infra Integration Validation Deployment Generation Int/Val tests Perf. Test Quality Check Dash Board Conf Mangt Extraction Generation (Preint) tests Quality Check Dash Board Labs Located Data Ingestion Processing Data Distribution Computing Center(s) EC level S/W conf mangt) OU-LE 3 KOM Meeting, London Production EMA March 22 -23, 2012 7

SDCs - Development Environment & Context Euclid Consortium • Tailored (i. e. simplified & SDCs - Development Environment & Context Euclid Consortium • Tailored (i. e. simplified & adapted) ECSS process should be applied • Linux platform • Model based Approach : Sys. ML • Scripting language : Python • Coding language : C/C++ • Distributed Development (cf. open source best practices) • Test driven approach • Iterative & Incremental process (Agile Approach) • Continuous S/W Integration • Strong configuration management • Automation (extraction, build, test, code quality check, …) • Dashboards (motivating !) • Online tools • Collaborative documentation OU-LE 3 KOM Meeting, London March 22 -23, 2012 8

OU / SDC-DEV – Agile Approach (1/2) Euclid Consortium • Iterative & Incremental development OU / SDC-DEV – Agile Approach (1/2) Euclid Consortium • Iterative & Incremental development – Sprints (e. g. monthly or less) : sub functions – Release (e. g. half year or less) => S/W available in early phases, ability to adapt the S/W OU-LE 3 KOM Meeting, London March 22 -23, 2012 9

OU / SDC-DEV – Agile Approach (2/2) Euclid Consortium • OU is “Product Owner”: OU / SDC-DEV – Agile Approach (2/2) Euclid Consortium • OU is “Product Owner”: – Backlog management : Prioritized list (e. g. priority, maturity, …) of pipeline “features”, – Definition of releases and sprints contents in coordination with SDC-DEV team (e. g. according to SDC-DEV capability), – Validation of pipeline with SDC-DEV team. • SDC-DEV team responsible of sprint & release dev. OU OU-LE 3 KOM Meeting, London SDC DEV March 22 -23, 2012 10

Continuous Integration Principles Euclid Consortium * Maintain a code repository * Automate the build Continuous Integration Principles Euclid Consortium * Maintain a code repository * Automate the build * Make the build self-testing * Everyone commits to the baseline every day * Every commit (to baseline) should be built * Keep the build fast * Test in a clone of the production environment * Make it easy to get the latest deliverables * Everyone can see the results of the latest build * Automate deployment OU-LE 3 KOM Meeting, London March 22 -23, 2012 11

SDCs - Development Phase Euclid Consortium • Pipeline Development in respect with SGS and SDCs - Development Phase Euclid Consortium • Pipeline Development in respect with SGS and System rules – ECSS tailoring (documentation, dev cycle) – UML modeling (common look & feel) – Pipeline I/F – IAL principles – Products, Data modeling with modeling tool – Quality rules (check should be as much as possible integrated and automated inside the collaborative platform) • Potential common S/W modules should be submitted at system level OU-LE 3 KOM Meeting, London March 22 -23, 2012 12

SDCs – Integration Phase Euclid Consortium • “target” pre-integration may be done at SDC-DEV SDCs – Integration Phase Euclid Consortium • “target” pre-integration may be done at SDC-DEV side on virtual machines being representative of target configuration. • Target Integration should be done on SDC-Prod infra. in a dedicated Integration « sand box » • Integration should be achieved in close cooperation of SDC-DEV, SDC-PROD and OU teams • “Technical” tests designed at SDC level • “Functional” tests designed at OU level • Integration tests output data should be accessible to above teams for validation and further investigation OU-LE 3 KOM Meeting, London March 22 -23, 2012 13

SDCs – Validation Phase Euclid Consortium • Validation should be done on SDC-Prod infra. SDCs – Validation Phase Euclid Consortium • Validation should be done on SDC-Prod infra. in a dedicated Validation « sand box » • Integration should be achieved in close cooperation of SDC-DEV, SDC-PROD, OU teams, and as needed SWG. • “Scientific” tests designed at OU/SWG level • Validation tests output data should be accessible to corresponding OU team and SWG for validation and in case of failure for further investigation. OU-LE 3 KOM Meeting, London March 22 -23, 2012 14

Euclid Consortium SDCs – Collaborative Development Platform • A Collaborative Development Platform should be Euclid Consortium SDCs – Collaborative Development Platform • A Collaborative Development Platform should be set up: – Aims : verify tools readiness, simulate dev cycle, may be used for SGS mock-up. – O/S : Linux – Languages : Python, C++ – Configuration : SVN – Generation : Maven – Test s : x. Unit – Integration : Jenkins – Quality : Sonar (with C++ plug-in) – IDE : Eclipse, Netbeans, vi, whatever… – Tracking : Jira, Redmine – Doc Gen. : Doxygen – Modeling : Sparx Enterprise Architect – Deployment : Iz. Pack OU-LE 3 KOM Meeting, London March 22 -23, 2012 15

Euclid Consortium SDCs - Collaborative Devlopment Platform Ec li So ps na r e Euclid Consortium SDCs - Collaborative Devlopment Platform Ec li So ps na r e Jir a Je nk ins OU-LE 3 KOM Meeting, London March 22 -23, 2012 & co 16

Euclid Consortium Tools and …. Rules But, tools are not sufficient, needs for : Euclid Consortium Tools and …. Rules But, tools are not sufficient, needs for : • Rules (e. g. naming conventions, formats, directory trees, …) • Guidelines • Code, Doc templates • Good practices • User support At SGS level OU-LE 3 KOM Meeting, London March 22 -23, 2012 17

Euclid Consortium SDCs – Typical Product Tree SDC-xx-DEV SDC-xx-PROD xxx pipeline Infrastructure yyy pipeline Euclid Consortium SDCs – Typical Product Tree SDC-xx-DEV SDC-xx-PROD xxx pipeline Infrastructure yyy pipeline (contribution) Integration Infrastructure Development Infrastructure Validation Infrastructure TL/IAL Plug-ins Production Infrastructure TL/IAL Connectors OU-LE 3 KOM Meeting, London March 22 -23, 2012 18

SDCs – Development Inputs • Euclid Consortium OU-xx outputs : – Pipeline workflow, breakdown, SDCs – Development Inputs • Euclid Consortium OU-xx outputs : – Pipeline workflow, breakdown, inputs & outputs and algorithms specifications, Use cases, … – Pipeline V&V kit (test data, expected results, documentation) – Optionally, prototype code : should be reengineered in order to fit SGS rule • Formal data modeling of all xx pipeline input (primary and secondary), output and intermediate data. • Code and Data from OU-SIM • SGS Level Documentation impacting the software development (RB, TS, DDF, DJF, MGT, PAF) to which the xx pipeline implementation should be compliant and/or should derive its own documentation. : – Architecture – Development principles – Common Rules & Tools – PA/QA – … • Architecture constraints from production infrastructure OU-LE 3 KOM Meeting, London March 22 -23, 2012 19

Euclid Consortium SDCs – Development Deliverables outputs • Pipeline product including Software executables, documented Euclid Consortium SDCs – Development Deliverables outputs • Pipeline product including Software executables, documented source code, under strong configuration control • pipeline documentation (RB, TS, DDF, DJF, MGT, PAF), in particular : – RB : Software system specifications (SSS), Interface requirements document (IRD) – TS : Software Requirements Specifications (SRS), Interface Control Document (ICD) – DDF : Software Design Document (SDD), Software Configuration File (SCF), Software Release Document (SRel. D), Software User Manual (SUM) including in particular extraction, deployment, installation and generation procedures. – DJF : Software V&V Plan (SVVP), Software Test Specifications (STS), Software Test Report (STR), Software problem reports and nonconformance reports (SPR & NCR) – MGT : Software Development Plan (SDP) – PAF : Software Product Assurance Plan (SPAP), traceability & coverage matrixes (TCM). OU-LE 3 KOM Meeting, London March 22 -23, 2012 20

Euclid Consortium SDCs - Integration Deliverables Outputs • Operational VIS pipeline on SDC-FR PROD Euclid Consortium SDCs - Integration Deliverables Outputs • Operational VIS pipeline on SDC-FR PROD infra. • Verification & Validation documentation : – Verification & Validation Plan (SVal. P) – Verification & Validation Tests Specification (SVal. S) – Verification & Validation Tests report (SVal. R) • Contribution with OU & PO/System team to system 2 by 2 tests report • Contribution to system E 2 E tests report OU-LE 3 KOM Meeting, London March 22 -23, 2012 21

SDCs - Operation Deliverables Outputs Euclid Consortium • Data products from pipeline • Production SDCs - Operation Deliverables Outputs Euclid Consortium • Data products from pipeline • Production reports : global metrics ( problems met during exploitation, performance : processing, storage, …) OU-LE 3 KOM Meeting, London March 22 -23, 2012 22

SDCs – Versions & Overall Planning Euclid Consortium SGS Global schedule Phase F Disposal SDCs – Versions & Overall Planning Euclid Consortium SGS Global schedule Phase F Disposal Proposed Schedule OU-LE 3 KOM Meeting, London March 22 -23, 2012 23