EVLA Data Processing PDR Overview Tim Cornwell NRAO

EVLA Data Processing PDR Overview Tim Cornwell, NRAO July 18 - 19, 2002 EVLA Data Processing PDR

EVLA: Data Management • EVLA has sub-contracted EVLA data management to NRAO Data Management group • End-to-end processing needs being addressed by DM End-to-end (e 2 e) project • Data reduction needs being addressed by DM AIPS++ project July 18 - 19, 2002 EVLA Data Processing PDR 2

Proposal Preparation and Submission Observation Preparation Observation Scheduling Monitor and Control System Antenna Feed Data Archive Image Pipeline Data Post. Processing CBE Correlator Receiver IF System Fiber Optics Transmission System NRAO Data Management Group (e 2 e project) Local Oscillator EVLA Project Canadian Partner Primary data flow Control and monitor flow July 18 - 19, 2002 Principal EVLA Subsystems EVLA Data Processing PDR 3

End-to-end goals • Streamline observer access to NRAO telescopes – End to end management from proposal to science – Cross-Observatory consistency • Greatly improve data products to users of NRAO radio telescopes – Provide original, calibrated, and auxiliary data, default images and processing scripts – Improve monitoring of instrument behavior • Greatly improve archive access – On-line access to archives of contemporary and historical images, surveys, catalogs, etc. – Technical and scientific data mining via web and NVO To reach these goals, initiated End-to-end Project in July 2001 July 18 - 19, 2002 EVLA Data Processing PDR 4

e 2 e requirements and scope • Extensive discussion of first pass scientific requirements with Scientific Working Group – Captured in e 2 e project book: http: //www. nrao. edu/e 2 e/documents/e 2 eprojectbook. doc – Proceeding on basis of current requirements – Description of workflow from proposal to observing script • Converted to high level architecture and data flow • Refine scientific requirements at end of phase 1 (July 2002) • Commit to design and scope at end of phase 2 (April 2003) – First e 2 e advisory group meeting ~ April 2003 • Spending ~ 15% of budget on planning – Good way to mitigate against risk July 18 - 19, 2002 EVLA Data Processing PDR 5

e 2 e development • Current staff – – – • John Benson, Tim Cornwell, Boyd Waters, Honglin Ye Lindsey Davis (IRAF, NOAO – to join in Sept, funded by ALMA), another later Doug Tody (IRAF, NOAO – to join in Sept, part of large NSF-funded collaboration) Use spiral development model – Develop in 9 month phases • • – – • Get requirements, plan, design, implement, test Review requirements, plan, design, implement, test…. . Five year development plan consisting of 7 phases Add new staff incrementally Three important principles 1. 2. 3. Keep it simple Reuse as much as possible Deliver new capabilities soon and often July 18 - 19, 2002 EVLA Data Processing PDR 6

e 2 e Architectural Diagrams July 18 - 19, 2002 EVLA Data Processing PDR 7

July 18 - 19, 2002 EVLA Data Processing PDR 8

July 18 - 19, 2002 EVLA Data Processing PDR 9

Overall e 2 e architecture Data flow July 18 - 19, 2002 EVLA Data Processing PDR 10

Telescopes and projects • • e 2 e will be retrofitted to all NRAO telescopes (GBT, VLA, VLBA) VLA – Putting archive on-line now, working towards pipeline processing • EVLA – Sub-contracted to deliver entire e 2 e system for EVLA (for 18 FTE-years) – Close interaction with EVLA project team at all levels • VLBA – Will start moving archive to disk after VLA archive – VLBA pipeline processing once AIPS++ can handle it • GBT – Designing archive facility for deployment in GBT early 2003 – Watching re-engineering of observing script generation • ALMA – – Sub-contracted to develop pipeline (framework only) and post-processing Start development July 2002 ALMA has own equivalent to all parts of e 2 e Trying for reuse if possible (e. g. Observation Scripting GUI from ALMA) July 18 - 19, 2002 EVLA Data Processing PDR 11

From NRAO to the National Virtual Observatory Q A Images, catalogs Referrals Data • Produce images and catalogs from well-documented pipeline processing • Images and catalogs available via NVO access tools e 2 e • All radio data stays within NRAO Services • Other wavebands have similar relationships to NVO • Built using web services and grid computing July 18 - 19, 2002 NRAO EVLA Data Processing PDR 12

Relationship of DM to ALMA project • ALMA has subcontracted development of offline processing and pipeline framework to NRAO • e 2 e: – Must deliver pipeline framework – No other re-use planned – Proposal submission, observation scripting will be different • AIPS++: – – ALMA processing requirements documents being finalized AIPS++ in baseline plan AIPS++/ALMA tests under way to test compatibility ALMA representative (Gianni Raffi) recently joined AIPS++ Executive Committee July 18 - 19, 2002 EVLA Data Processing PDR 13

e 2 e timescales • Customer requirements – – – • First cycle of development (ended July 15, 2002) – – • GBT archive facility Thorough testing of archive and pipeline for VLA Development of prototype observation scripting and scheduling First advisory committee meeting End of overall generic development (2006) – – • Prototyped VLA archive and pipeline software and facility Started loading VLA archive to disk Improved support for VLA/VLBA calibrator database Design for proposal submission and management Second cycle of development (ends in Q 2 2003) – – • EVLA PDR process in 2002, Working M&C by early 2004, Shared risk science 2007 ALMA development, Phase II starts this year, runs to 2006 GBT archive facility by end of proprietary period (early 2003) NSF funding for archive work Sept 2001 – Sept 2003 Project book (http: //www. nrao. edu/e 2 e) contains scientific requirements as currently understood Working archives, pipelines, ancillary software for VLA, VLBA, GBT First generation for EVLA, ALMA Move onto EVLA and ALMA specific development (2006+) July 18 - 19, 2002 EVLA Data Processing PDR 14

EVLA critical dates July 18 - 19, 2002 EVLA Data Processing PDR 15

Costing, schedule, deliverables, etc. • Plan is to develop design in all e 2 e areas to level required to cost the project by end of development cycle 2 (April 2003) • At that point, e 2 e commits to requirements, costing, schedule, deliverables • Scope adjustments will be made at beginning of development cycles as agreed with EVLA July 18 - 19, 2002 EVLA Data Processing PDR 16

e 2 e resources • ALMA numbers estimated by ALMA computing management • Seem to be in line with other ground based projects but considerably less than space based • e 2 e numbers based upon straw man designs, reuse • e 2 e scope will be adjusted to fit resources (~ 55 FTE-years) • Neither constitute a detailed bottom-up derivation of resources from requirements July 18 - 19, 2002 EVLA Data Processing PDR 17

De-scoping options • De-scoping occurs first within toolkits via priorities set by EVLA project – Potentially large de-scoping available here • Next toolkits can be removed • e 2 e is committed to provide Pipeline for ALMA – Pipeline requires Observation Scripting, Observation Scheduling, Archive • Core architecture can survive removal of: – Telescope Simulation – Observation Evaluation – Remote Observing • Spiral development allows these de-scopes to be made incrementally (at the beginning of each development cycle) July 18 - 19, 2002 EVLA Data Processing PDR 18

AIPS++ resources • Expect roughly the same level of effort from AIPS++ on EVLA as on VLA currently • Total effort ~ 10 FTE-years from 2003 to 2009 • Addressing EVLA-specific processing issues July 18 - 19, 2002 EVLA Data Processing PDR 19

EVLA-specific post processing • Mostly well-understood and in place – AIPS++ package: can reduce VLA data end-to-end – BUT final requirements yet to be set • EVLA-specific areas requiring more development – New modes of processing (next slide) – Very large data volumes • Automated flagging schemes • Performance issues – Ensure that AIPS++ is efficient and fast enough (compare to AIPS) • AIPS++/AIPS speed ratio ~ 1 +1/-0. 5 (with some outliers!) – Develop parallelized applications (e. g. imaging, calibration) • Well in progress in collaboration with NCSA – Develop location independent computing (a. k. a. Grid computing) • e. g. transparent access to archive and pipelines from remote locations July 18 - 19, 2002 EVLA Data Processing PDR 20

Examples of EVLA hard processing problems Fast-slew mosaicing ~10 ms data sampling rate. Remove sliding primary beam. Full bandwidth synthesis Deconvolve wide bandwidths while accounting for spectral index, polarization, rotation measures, opacity, etc. Full-beam high-fidelity polarization imaging Correction of time- and angle-dependent beam polarization. High fidelity imaging Image and deconvolve at ~ 107. Currently about ~ 100 away from this in best possible cases. Wide-angle full-beam imaging Huge images, fast data sampling rates, many imaging facets to accommodate non-coplanar baselines RFI mitigation Removal of RFI post-correlation – requires high data rates July 18 - 19, 2002 EVLA Data Processing PDR 21

e 2 e status July 18 - 19, 2002 EVLA Data Processing PDR 22

Risks • Creeping scope – Requires project discipline – e. g. scientific requirements for post-processing soon • Lack of engagement by scientific staff – Work with DM Project Scientist (Dale Frail), DMSWG • Observation scripting too hard – Develop incrementally • Pipeline processing cannot be made to work for significant fraction of observations – Prototype on VLA: will require some changes to current practices • Archive = Operational morass – Need automation and management staff soon • Repeat of AIPS++ July 18 - 19, 2002 EVLA Data Processing PDR 23

Lessons learned in AIPS++ project • Software development: – – – • Package deployment: – – – – • Start new software development projects with realistic expectations Control scope: initial requirements were developed without a reliable costing process Management of distributed software projects is especially demanding Establish firm staffing commitments Continual refinement of processes important: moved to spiral development Demonstrate scientific completeness: establishing threads of completeness by matching representative data to reduction scripts User testing is vital: formed active, large Observatory-wide test group Robustness: identifying and fixing defects as submitted Performance must be regularly monitored: established benchmark suite, scheduled regular profiling, targeting known cases of poor performance User interface design is very demanding: conducted one-on-one testing and group surveys Documentation forms a gateway to the package: enlisted help of scientists in writing documentation Training is best way to introduce new users to AIPS++: presenting tutorials to small groups Lessons learned applied across the Observatory, ALMA, e 2 e July 18 - 19, 2002 EVLA Data Processing PDR 24

Specific changes adopted by e 2 e • Spiral model – Short development cycle – Deliver early and often • Involvement of scientists – – – Set specifications at beginning of cycle 1 AOC scientists tested and advised on Calibrator Source Toolkit Will review and change specifications at beginning of cycle 2 Dale Frail will be DM Project Scientist Will be involved in pipeline development, testing of archive and proposal handling during cycle 2 – Advisory Group meeting at end of cycle 2 • Commit to requirements, plan, costing, schedule – Design and development phase (first two cycles) ending in April 2003 – Schedule, etc. then set July 18 - 19, 2002 EVLA Data Processing PDR 25