ne W USe of an old correlator Arpad

ne. W USe of an old correlator Arpad Szomoru Joint Institute for VLBI in Europe Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Mark IV EVN Correlator • Developed by international consortium: EVN institutes, MIT • Officially inaugurated October 1998 • Comparable correlators in use at Haystack Obs. , US Naval Obs. , MPIf. R, JIVE • Correlator boards in use at WSRT, SMA Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Basic specifications Input: • • • No. of telescopes (N) 16 Array Observing frequencies: 329 MHz – 22 GHz Data bandwidth: 128 MHz per polarization (Right and Left -hand circular), divided into 8 bands. Channel bandwidths 0. 5 MHz to 16 MHz. Data input rate: 1 Gbps per telescope • • • Integration time 1/4 s (will become 1/32 s with PCInt) 2048 spectral channels per baseline/band/polarization Data Output rate 6 MB/s (will become 80 MB/s with PCInt) Output: Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Field of View limitation VLBI Fo. V x 100 • Limited by tint • Time smearing • Shorter integrations • Enable wide field surveys • Study μJy sources • Discriminate AGNs • But, enormous increase of output data volume. . 6 arcmin [FWHP] Effelsberg beam Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

EVN Mk. IV Correlator limits • Integration time • Cycle time of 0. 015 s (actually, 1/64 th of a second) • Spectral resolution • 131072 complex lags per readout = 65536 spectral points per readout • Divided over 32 products leads to 2048 spectral channels per product PCINT: • Short for Post Correlator Integrator • Capture the full output of the EVN Mk. IV correlator to disk • Need to replace output datapath Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

The PCInt project • High speed readout of the correlator was already prepared • Via DSP powered serial port • Need hardware and software to enable this • Receiving end of serial port • Gbit ethernet for transfer from correlator rack to data collection host • Fast disk subsystem in order to support 160 MByte/s (parallel RAID arrays in a Storage Area Network) Harro Verkouter Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

RT System VME High Speed Serial C 40 COMM PCI Ethernet Card SBC (x 2) Correlator rack (x 4) 100 TX 1000 FX (x 8) Switch CCC DDD (xn) Fibre Channel EEE (xk) FC Switch Raid Array (xm) Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE Current situation Correlator Board (x 8)

RT System Correlator Board (x 8) C 40 COMM PCI Ethernet Card SBC (x 2) Correlator rack (x 4) 100 TX 1000 FX (x 8) Switch CCC DDD (xn) Fibre Channel EEE (xk) FC Switch Raid Array (xm) Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE Phase 0 High Speed Serial VME

RT System High Speed Serial VME C 40 COMM PCI Ethernet Card SBC (x 2) Correlator rack (x 4) 100 TX 1000 FX (x 8) Switch CCC DDD (xn) Fibre Channel EEE (xk) FC Switch Raid Array (xm) Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE Phase 1 Correlator Board (x 8)

RT System Correlator Board (x 8) C 40 COMM PCI Ethernet Card SBC (x 2) Correlator rack (x 4) 100 TX 1000 FX (x 8) Switch CCC DDD (xn) 1000 TX EEE (xk) Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE Raid Array (xn) Phase 2 High Speed Serial VME

Post-processing issues • Huge data volumes 1 hour @160 MByte/s equals 560 GBytes of data • Use a cluster of nodes • Need automated processing • Use a processing pipeline Achieved 1/16 s sampling, at 24 MB/s data output Users seem to be ready for 800 GB data-sets… Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Price recording media ($/GB) Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Data Acquisition • Move from tape to disk recording Disk based recording • Reliability • Cost • Bandwidth • Efficiency • e-VLBI: the next step • No consumables • Higher bandwidth • Fast turn-around • To. O support Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE e-VLBI using fiber

Why e-VLBI ? • Reliability – real-time feedback to the telescopes • Logistics – No media management • Sensitivity – sustained data rates >> 1 Gbps possible… • Rapid science results: • Geodesy (Earth rotation rate) • Precision spacecraft navigation • Transient phenomena… GRBs, SNe etc. Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Why e-VLBI (cont) ? • Target of Opportunity (To. O) capability: • Dominated by VLBA currently… • Reliability & Logistics e-VLBI • Sensitivity e-VLBI • Rapid science e-VLBI • Rapid publication e-VLBI • Optimal observing strategy (obs. freq. , calibrators, telescope array) • LOFAR Transients etc. To. Os may become much more common e -VLBI. Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

e-VLBI Proof-of-Concept Project • • • • DANTE/GÉANT Pan-European Network SURFnet Dutch NREN GARR Italian NREN UKERNA UK NREN PSNC Polish NREN DFN German NREN KTHNOC/NORDUnet Nordic NREN Manchester University Network application software JIVE EVN Correlator Westerbork telescope Netherlands Onsala Space Observatory Sweden MRO Finland MPIf. R Germany Jodrell Bank UK TCf. A Poland CNR IRA Italy Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

GÉANT: Access of NRENs to GÉANT NL DE BE 2. 5 HU IT FR 1. 2 G GR CZ AT G 62 UK CH SE 2 M 0 GEANT ES PL 34 M HR RO EVN telescope LV BG Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE 45 M CY LT PT 310 M 55 SI M IE 1 LU IL SK EE

POC results • Demonstration of feasibility • Identification of problems • Has led to closer ties with networking community and generated political interest • Has laid the foundation for the next step forward (EXPRe. S): • I 3 proposal to the EC (Communication & Network Development Call) • Ranked first out of 43 proposals; nearly fully funded to an amount of 3. 9 MEuro. Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

EXPRe. S – major aims: • Making e-VLBI an operational astronomical instrument: • • 16 telescopes connected to JIVE at 1 Gbps Robust real-time e-VLBI operations Transparent inclusion of e-MERLIN antennas within e-EVN Target of Opportunity Capability • Future developments in e-VLBI • >> 1 Gbps data transfer rates, • extended LOFAR etc. • distributed software correlation. Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Expanding the e-VLBI Network Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Network testing • Use existing protocols on currently available hardware • TCP maximal reliability • Not really required • Sensitive to congestion • Lot of fine-tuning necessary • And possible • UDP connectionless • Unaccountable • Tailor made protocols? • Lambda switching • Internet weather • Hard to quantify • Hard to pinpoint bottlenecks Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Network testing (2) • February 2005: network transfer test (BWCTL) employing various network monitoring tools involving Jb, Cm, On, Tr, Bologna and JIVE Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

First real-time e. VLBI Image, 3 telescope observation of gravitational lens, May 2004 First e. VLBI science observation, OH masers around IRC 10420, Richards et al, Oct 2004 First broadband e. VLBI science, detection of the Hypernova SN 2001 em, Paragi et al, astro-ph/0505468 Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

• First open e-EVN Call for Proposals (March 2006) • First Target of Opportunity Observations (May 2006, Cygnus X-3), Tudose et al. (in prep)… Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Current status • Regular science runs at 128 Mbps with 6 European stations (24 hours) • Arecibo sometimes participates at 32 Mbps • Fringes from all European stations at 256 Mbps have been demonstrated, and, • on single baselines, 512 Mbps Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Issues, Developments Convincing a correlator designed for tape technology to become real-time. . Operational improvements: • Robustness • Reliability • Speed • Ease • Bandwidth • Station feedback Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Ongoing New control computers (Solaris AMD servers) • Cut down on (re-)start time • Powerful code development platform • Tightening up of existing code Other hardware upgrades: • Upgrade existing connectivity from 6*1 Gbps to 16*1 Gbps (lightpaths) • SX optics (fibres + NICs) • Replacement of SU functionality: Mark 5 A→B: motherboards, memory, power supplies, serial links. Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE

Conclusions • e-VLBI is changing the nature of VLBI • Fast response, To. O capability • better quality control, rapid data delivery • New science, higher bandwidths • Large fields of view • Will allow the study of μJy sources • Or many masers over a large star formation region • Data archive will contain millions of weak sources • EXPRe. S will realize an operational e-VLBI network distributed across 1000’s km – a true pathfinder for SKA. Next Gen. Correlator Workshop, Groningen, June 2006, A. Szomoru, JIVE