SA 1 Production e-VLBI service Arpad Szomoru JIVE

SA 1: Production e-VLBI service Arpad Szomoru JIVE

Outline • • 2008 May 30 e-VLBI science/test runs Operational improvements Global e-VLBI: the demonstrations Hard- and software developments EXPRe. S Annual Review: SA 1 2

e-VLBI science/test runs • 1 Target of Opportunity project (target: nearby supernova) • 1 response to a triggered proposal (target: accreting X-ray binary in a special X-ray state) • 2 exploratory projects (both targets were candidate radio counterparts to so far unidentified gamma-ray source) • 1 Spectral line project (targets: candidate MASER sources) • October deadline only had one triggered proposal • Lack of triggers meant no science runs until March 2009 • . . . but, available time was used very well • Different proposal/selection mechanism clearly needed • 3 rd year already much more productive 2008 May 30 EXPRe. S Annual Review: SA 1 3

e-VLBI: operational improvements • Overall improvement in second year: • Huge increase in robustness, reliability and user-friendliness • Not limited to e-operations • Increase of operational data rate from 256 to 512 Mbps • First experiments at ~1 Gbps (with European telescopes) • Demonstrations of global e-VLBI with Chinese and Australian telescopes • Solution of long-haul problems • Inclusion of Effelsberg, Metsahovi, Shanghai, Arecibo (again), TIGO 2008 May 30 EXPRe. S Annual Review: SA 1 4

e-VLBI: data rate improvements 2008 May 30 EXPRe. S Annual Review: SA 1 5

Towards global e-VLBI • Long-haul high-bandwidth data transport: • TCP Reno on old linux kernels clearly inadequate • Modern TCP stacks much better, UDP even better, but impossible without modification of Mark 5 A control code • And can be hostile to other users on open network • Stability of original Mark 5 code a serious issue 2008 May 30 EXPRe. S Annual Review: SA 1 6

Global e-VLBI: hacking time • With several high-profile demos coming up: time to take matters in our own hands • First step: re-enable UDP in Mark 5 A code • abysmal performance. . . • Incorporate VLBI-UDP in Mark 5 code, in collaboration with colleagues from UMan • Simplified version developed at JIVE, used in both APAN and EXPRe. S-Oz demo • Complete re-write of Mark 5 control software, with rigorous thread control, enabling packet dropping, re-ordering, padding • Used successfully to transfer data at near- line-speed from European telescopes, from Arecibo and, recently, from South Africa and South America • Large development effort, but has paid off in a big way 2008 May 30 EXPRe. S Annual Review: SA 1 7

The road to China: the demo at Xi’An 2008 May 30 EXPRe. S Annual Review: SA 1 8

The APAN demo • Needed: • Reliable transport from standard Mark 5 at Shanghai (ancient Linux kernel) • Physical connection between two noncommunicating Chinese NRENs • Some way to bypass the Great Chinese Firewall • Interface between correlator and Australian telescope control systems • On-the-fly translation from LBA to Mark 5 B format • What could possibly go wrong. . . 2008 May 30 EXPRe. S Annual Review: SA 1 9

APAN results • Opened up Siberia connection • Excellent PR • Longest (e) baseline ever 2008 May 30 EXPRe. S Annual Review: SA 1 10

The EXPRe. S-Oz demo • Three Australian telescopes (Mopra, ATCA, Parkes) • 12 hours at 512 Mbps without a hitch • Beautiful detection of SN 1987 a 2008 May 30 EXPRe. S Annual Review: SA 1 11

Correlator upgrades • Network upgrade: • All internal JIVE and e-VLBI traffic now through own routers (HP 5412 zl and HP 3500 yl-48 Pro. Curve) • Found quite a few of (serious) bugs in HP equipment • So far, one bug fix out of 5 active cases • Multiple 1 -Gbps UDP streams not foreseen by HP engineers? • Dedicated network monitoring platform alongside routers • Removal of dependencies on ASTRON network • Upgrade of Mark 5 hardware (motherboards, memory, power supplies) completed • New data acquisition platform (Solaris AMD server, fully interchangeable with redundant control computers 2008 May 30 EXPRe. S Annual Review: SA 1 12

New modes, feedback and streamlining • Completed: • General speedup of correlator control code • Real-time Mark 5 debug info feedback to stations • Re-connecting of individual stations without job re-start • Space-craft tracking mode • Use of Westerbork synthesis data to improve e-VLBI calibration and target selection • Network monitoring tools • Currently under development: • Adaptive observing • On-the-fly fringe fitting • Real-time station-log processing, display of Tsys, status information • Mixed-mode e-VLBI • Integration of Mark 5 B • 1024 Mbps e-VLBI (and more. . ) 2008 May 30 EXPRe. S Annual Review: SA 1 13

New network monitoring tools. . . 2008 May 30 EXPRe. S Annual Review: SA 1 14

And new telescopes. . . 2008 May 30 EXPRe. S Annual Review: SA 1 15

In summary • Guaranteed high bandwidth through use of an aggressive protocol *and* reliable software • Lightpath technology offers clear advantages; routed connections need negotiations • Convincingly demonstrated that it is possible to overcome long-haul connectivity problems • Simplicity paid off: trivial to connect new EVN stations once they come online 2008 May 30 EXPRe. S Annual Review: SA 1 16

2008 May 30 EXPRe. S Annual Review: SA 1 17

2008 May 30 EXPRe. S Annual Review: SA 1 18

SA 1: state of deliverables • A total of 15 deliverables were due by month 24, 11 at JIVE and 4 at the University of Manchester • 8 out of 11 JIVE deliverables were completed • Most of the work on the remaining 3 was finished • 3 deliverables that were not due until the last year of EXPRe. S completed well ahead of time were • The 4 Uni. Man deliverables were delayed because of the delay in the delivery of the e-Merlin correlator. Most of the IBOB design work has been completed, but tests await the arrival of the first I/O boards, in the summer of 2008 May 30 EXPRe. S Annual Review: SA 1 19

Outsourcing • Three workpackages were outsourced to MIT: • SA 1. 24: Mark 5 A code modifications • Working version available (though no “official” release yet) • SA 1. 25: Mark 5 B code modifications • Dimino (sending) side largely done in-house • Domino (receiving) side developed at MIT, debugging in progress • Extensive engineering support in integration of Mark 5 B • SA 1. 20: Improved network applications • done in-house, with MIT support mainly 2008 May 30 EXPRe. S Annual Review: SA 1 20

SA 1: the next 18 months • The focus of SA 1 for the next 18 months will be on • Adaptive scheduling • Further integration of Mark 5 B in EVN operations • Streamlining operational data rate adjustments • Further improving the robustness and flexibility of the correlation process • Mixed-mode operations • Increasing the data rate beyond 1 Gbps 2008 May 30 EXPRe. S Annual Review: SA 1 21

Specific tasks • Fast/adaptive (re-)scheduling • Transform EVN into a truly flexible instrument • Based on automated download and execution of modified observing schedules - possibly on-the-fly • Reactions from stations vary from “excellent idea” to “over our dead bodies” • But will further improve reliability and responsiveness of EVN • All bits tested and in place, full operational test is planned 2008 May 30 EXPRe. S Annual Review: SA 1 22

Specific tasks (2) • Data rate adjustments • Packet dropping: • Drop packets at sending side (while preserving headers!) • Pad datastream at receiving side with dummy data • Channel dropping: • Selectively drop frequency channels at receiving side • High demand on cpu, need hyperthreading, right kernels, dual cpu’s • Mixed configurations: • Use different configurations at different stations (limited to 32, 64, 128 etc Mbps) 2008 May 30 EXPRe. S Annual Review: SA 1 23

Specific tasks (3) • Mixed-mode operations: • Enable all possible combinations of recording (at stations and/or at correlator), real-time and delayed data transfer • Make best use of complete EVN network, including stations that have less-than optimal connectivity 2008 May 30 EXPRe. S Annual Review: SA 1 24

Specific tasks (4) • Beyond 1 Gbps • 1024 Mbps will simply not fit on 1 Gbps. . . • Westerbork: use of DWDM equipment, dual 1 -Gbps connection, channel bonding • Could be used to Jodrell Bank as well (dual 1 -Gbps lightpaths to JIVE) • Onsala, Effelsberg: share 10 -Gbps connections with e. LOFAR, via VLAN (nearly in place) • Next, Torun? • Most sensitive telescopes in Europe soon at a full 1024 Mbps • And tests ongoing at 4 Gbps, from Onsala 2008 May 30 EXPRe. S Annual Review: SA 1 25

And soon to come. . . • Yebes first light (any moment now) • Hartebeesthoek: demo last month at 64 Mbps, S-African universities expect to be connected to Europe at 10 Gbps in 2009 • Arecibo: recent tests at ~512 Mbps • St Croix? • Urumqi: 622 Mbps connection (still in rumor phase) • Japan: talks ongoing about combining VERA telescopes with EVN in e-VLBI pilot project • Australia-China-Japan e-VLBI demo upcoming (at e-VLBI workshop in Shanghai) 2008 May 30 EXPRe. S Annual Review: SA 1 26

2008 May 30 EXPRe. S Annual Review: SA 1 27

SA 1 - Past 12 month overview SA 1 Deliverables D# A AD# Descrip Plan Actual 11 SA 1. 02 Job preparation utilities 16 27 SA 1. 04 e. MERLIN VSI interfaces design 13) 22 28 SA 1. 05 Network Protocol decision: 16 45 SA 1. 09 Tests using local Jodrell Bank home e-MERLIN telescope 14) 22 51 SA 1. 13 Tested software for operational improvements 18 52 SA 1. 14 Test using remote e-MERLIN telescope 17) 24 54 SA 1. 15 VSI Interfaces 16) 23 16 4 71 SA 1. 17 Flexible local GE network 4) 18 19 4 84 SA 1. 18 Network monitoring tools 20) 30 24 4 85 SA 1. 19 Multiple e-MERLIN telescope tests 24 ? 95 SA 1. 2 Improved network applications 3) 17 2 96 SA 1. 21 Monitoring user interfaces 30 7 4 124 SA 1. 22 Mark 5 hardware upgrade 16 20 4 126 SA 1. 23 Use of WSRT synthesis data for e-VLBI calibration 17 22 4 127 SA 1. 24 Mark 5 A code modifications 17 2 128 SA 1. 25 Mark 5 B code modifications 17 2 136 SA 1. 26 Investigating a 1024 M sub-array 20 23 4 148 SA 1. 27 Space craft tracking correlator mode 27 20 4 161 SA 1. 31 Removing/adding stations from the correlation process on the fly 34 2008 May 30 EXPRe. S Annual Review: SA 1 16 Status 4 2 20 4 ? 18 4 ? 28