RFI Mitigation Techniques at the ATA Garrett Karto

RFI Mitigation Techniques at the ATA Garrett “Karto” Keating RFI 2010 – Groningen, Netherlands March 31 st, 2010

A Quick (Re)Introduction 20 ft/6. 1 m Primary Wideband (0. 5 to 11 GHz) Currently 4 independent IFs and 3 independent FX 64 -input correlators Currently 42 Dishes Planned to move to 350 dishes, producing 61, 425 unique baselines Data volume is destined to become HUGE

Welcome to RAPID Rapid Automated Processing and Imaging of Data. Flag, Calibrate, Image, Repeat.

RAPID and ARTIS • “Offline” data reduction package for ATA data • Development began in January 2008 • “Online” data reduction package for ATA data • Offshoot of the RAPID project • First light in March 2008 • In regular use since Feb 2009

The truth will set you free… I am not a software/ computer engineer A “real” engineer would program in a “real” language “Effective” troubleshooting I “program” using shell scripts (awk/grep/sed) wrapped around MIRIAD

Mac. Gyver would be proud “Like building a radio telescope from a Swiss army knife and duct tape” –Anonymous ATA Engineer Limitations arise from programming choices: Limited math support Messy code Slow processing The impact of these limitations can be minimized!

Overall Processing Philosophy Assume Nothing • Nothing is every truly “static” – LNAs, antennas, RFI, correlators and even objects in the sky are all dynamic Pragmatic Processing • Processing choices must maximize time utility “Fallback” Processing • Assume tasks will fail to do everything correctly • Allow for recovery at each processing stage (RFI/Calibration/Imaging)

System Overview Images USER Master Obs Antennas Correlator ARTIS Catcher Flag Archive Calibrate Image

The RFI “cube” Time Long Medium Short Intensity Narrow Multichannel Broadband Ba ro n St ra g te k ea W od e nd w id th • Most RFI cases can be adequately described by its length, bandwidth and intensity • Each case requires a unique strategy and multiple layers of mitigation • Not to be confused with Rubik’s cube M “Axes of RFI”

Power “Counts” Spectral Occupancy Frequency

Spectral Occupancy

Spectral Occupancy x 100 Scale 0. 2 s 0. 1 s

Spectral Occupancy Data is broken up into temporal windows - the more spectra processed, the better/deeper the RFI processing can go With enough spectra, extremely weak RFI can be identified and excised. Bad channels and surrounding channels are flagged.

Benefits No gains solutions required • Most spectra can be processing without having solved for the gains solution first Results can be easily combined • Spectral occupancy results can be combined across antennas and correlator dumps, or “windowed” to limit processing to only select antennas, baselines, etc. Processing is fast/computationally inexpensive • Achieves an 8: 1 observing to processing ratio at the ATA

Drawbacks SNR Limitations • Results dependant on noise-to-bandpass feature ratio • Normally becomes an issue with bright objects • Can be corrected by normalizing datasets • Can also be corrected with a good gains solution and sky model

Drawbacks Is it really RFI? • Results can confuse wanted and unwanted RF emissions • Can be solved by creating “good frequency range” masks when flagging • Can also be corrected for during the imaging stage

Drawbacks Results require a large number of samples • The smaller the array, the higher the required dump rate or the lower the sigma threshold Multiple iterations sometimes required • More powerful RFI may need to be culled first before reaching weaker RFI Trouble with “burst” RFI • RFI with an extremely short duration not likely to be picked up as consistently

Imaginary Threshold RFI Removal Real

Benefits Fast with calibrated data • Hours of data can be processed with this method in a matter of minutes Effective against “burst” RFI • Good at catching RFI left behind by spectral occupancy flagging No iterative processing generally required • With calibrated data, generally one pass is adequate

Drawbacks Slower with uncalibrated data • When used in conjunction with gains-solving processes, normally a 1: 2 observing to processing ratio is achieved Iterative processing with uncalibrated data • Process must be repeated after a new gains solution is built Vulnerable against weaker RFI • RFI below the noise threshold difficult to catch

WRATH RFI Removal Continuum Image Channel by channel

WRATH RFI Removal Pre-WRATH flagging stands as the “last guard” against RFI Dynamic range and fidelity in images can easily double following use of the WRATH mechanism Post-WRATH

Benefits Excellent “last guard” • Can generally catch the weakest of RFI Fast processing, not dependent on data size • Processing scales with number of spectral channels, relatively invariant to the size of the dataset Robust against calibration errors • Gains-solution errors usually affect all channels, thus don’t bias results

Who needs HI?

Who needs HI? Spectral line problems • Spectral line features can (again) be misrecognized as RFI • Not a lost cause! • Spectral line channels can be masked a priori • Spectral line features should still image with “better” deconvolution, RFI should not

Drawbacks Relatively slow for small datasets • Again, processing time is primarily proportional to number of channels Vulnerable against “burst” RFI • RFI with an extremely short duration not likely to be picked up “Baby and bathwater” dilemma • Highest risk for throwing away good data, since imaging artifacts may only be due to a single datapoint

Summary Current system is robust against most RFI • Still some trouble with transient/”burst” RFI, but future upgrades to fix that Processing able to keep up with observing • As processing speed increases, more processing tasks and choices can be made to render better images • Processing is highly scalable Good astronomer acceptance • Most astronomers using ATA data use the RAPID package, particularly the RFI flagging routines

Source code available at svn. hcro. org/mmm/karto/RAPIDBeta karto@hcro. org Office number: 1 -530 -335 -2364 Allen Telescope Array/Hat Creek Radio Observatory