A Comparison of Microarray Platforms NUS IMS

A Comparison of Microarray Platforms NUS – IMS Workshop 7 January 2004 Darlene Goldstein

Talk Outline • Bioinformatics Core Facility at ISREC • Purpose of study • Platform technologies and study design • Comparisons between platforms • Conclusions and study completion

BCF: What is it ? DAF NCCR biomedical BCF microarray research NCCR biomedical DAFL & BIOINF BCF bioinf. research biostatistics. EPFL • ISREC-based, supported by the NCCR for molecular oncology, member group of the SIB • Created by the NCCR molecular oncology to assist its DAF (which is now absorbed into the DAFL) and its microarray users in their biomedical research • A group devoted to the bioinformatics and statistical aspects of gene expression research, in particular to the analysis of data generated with microarray technologies

BCF: Main Components • Technical Support – advice in experimental design and data analysis – production, control, development of spotted arrays – processing of microarray data, quality assessment • Education – practical training through classes / workshops • Collaboration – statistical data analysis of research projects • Research & Development – development / testing tools & methods

Platform Comparison Study • Purpose – to assess accuracy and reproducibility of different gene expression platforms – to compare features of different measurement types – to understand the system (important for normalization and downstream analysis) • Impact – practical advice to DAF(L) and to NCCR microarray users – benefit to wider scientific community, especially if possible to somehow combine results across array types

Platforms and Study Design • Platforms – Affymetrix Gene. Chips, high-density short oligo arrays – Agilent long oligo arrays – in-house spotted c. DNA arrays – MPSS (massively parallel signature sequencing, a digital gene expression technology patented by Lynx); in collaboration with the Ludwig Institute for Cancer Research; originally intended as ‘gold standard’ • Basic Design – 3 replicate measurements for two m. RNAs (human placenta and testis) – dye swap for two-color systems (Agilent, c. DNA) – 2 to 3 million tags sequenced for MPSS

Methods • Experimental Method (as recommended by ‘specialists’): – Affymetrix: Biozentrum Basel – Agilent: Institut Goustav Roussy, Paris – Spotted c. DNA arrays: Otto Hagenbuechle's group (DAF, now DAFL) – MPSS: Lynx (California), Victor Jongeneel's group (LICR) – q. RT-PCR followup (~ 250 genes), Robert Lyle, Patrick Descombes (Uni. GE) • Expression Quantification as recommended by ‘specialists’ (above), but : RMA for Affymetrix

Spotted c. DNA arrays Human 10 k Array 8 x 4 subarrays

Affymetrix Gene. Chips Image of hybridized array

MPSS • Uses microbeads with ~100 k identical DNA molecules attached • Captures and identifies transcript sequences of expressed genes by counting the number of individual m. RNA molecules representing each gene (information from Lynx web site) • Individual m. RNAs are identified through generated 17 to 20 -base signature sequence • Can use without organism sequence information • ‘MPSS can accurately quantify transcripts as low as 5 transcripts per million (tpm) to above 50, 000 tpm’

Other comparison studies (I) • Yuen et al. 2002; Nuc. Acids Res. 30(10): e 48 – Affy MGU-74 A, c. DNA; cell lines; q. RT-PCR 47 genes – both arrays sensitive (TP) and specific (TN) at identifying regulated transcripts – found comparable rank-order of gene regulation, but only modest correlation in fold-change – both array types biased downwards (FC under-estimated compared to q. RT-PCR) • Evans et al. 2002; Eur. J. Neuroscience 16: 409 -413 – Affy RG-U 34 A, SAGE to detect brain transcripts; 43 rat hippocampi; evaluation based on 1000 transcripts – ~55% low, ~90% high abundance transcripts detected

Other comparison studies (II) • Li et al. 2002; Toxicological Sciences 69: 383 -390 – Affy Hu. Gene FL, HGU-95 Av 2, Incyte. Genomics Uni. Gem. V 2. 0 (‘long c. DNA’); drug-treated cell lines at 8 h and 24 h; q. RT-PCR 9 genes – cross-hyb contributed to platform discrepancies – found Affy ‘more reliable’ (sensitive) • Kuo et al. 2002; Bioinformatics 18: 405 -412 – Affy HU 6800, c. DNA, publicly available data on NCI 60; 2895 genes – found low correlation between measurements (but no control over lab procedures – different groups had performed the original studies)

Other comparison studies (III) • Barczak et al. 2003; Genome Res. 13: 1775 -1785 – 2 versions of spotted long oligo (Operon), Affy HGU-95 Av 2; cell lines; 7344 genes – this large-scale analysis found strong correlations between relative expression measurements – similar results for amplified and unamplified targets • Tan et al. 2003; Nuc. Acids Res. 31: 5676 -5684 – Agilent Human 1, Affy HGU-95 Av 2, Amersham Codelink Uni. Set Human I (30 -mers); cell lines in serum-rich medium and 24 h after serum removal; 2009 genes – modest correlations – little overlap in genes called DE – best agreement on DE calls (varying criteria) only 21% • comparison studies by other groups world-wide are also in progress

Comparison Principle • Cross-platform gene matching done through the trome database of transcripts (constructed with the Transcriptome Analyzer program tromer) • Use only those genes we classify as ‘reliably mapped’ between platforms (~2500 genes); we have not (yet) looked at probe(set)s that could not be well-mapped to known transcripts • ‘Peak technical performance’ : this is a case study, not a systematic study; does not take into account normal user variation, other m. RNAs, etc. • Comparison based on M (log ratio) and A (average log intensity) • Unfortunately, accuracy cannot be properly assessed, as true M values are not known

c. DNA array Performance

MA plots (examples) Affy U 133 A range background NCCR h 10 kd Agilent

|M| (putative effect) densities

(Difference in M) vs. A: reproducibility Affy U 133 A y = difference in M x = average A NCCR h 10 k Agilent h 1 A

D (error) densities

Gene Matching Affy Agilent 5977 5099 5797 3365 435 549 2514 NCCR Probe(sets) / genes 18325 Agilent h 1 A 15688 24808 Affy U 133 A 14876 7812 NCCR h 10 k 6853

Gene matching also with MPSS Affy Agilent - - 2494 - NCCR 2494 4060 2869 2685 Tromer clusters Affy probesets Agilent probes NCCR clones

Concordance in M density plots (I) Agilent Affy NCCR

Concordance in M density plots (II) Agilent Affy NCCR

Difficulty in comparing to MPSS ratios

MPSS difficulties, another illustration

Correlations first quartile (25% least frequent RNAs) fourth quartile (25% most frequent RNAs)

Agreement: top up 200 (placenta) Affy Agilent 30 40 34 48 96 26 44 NCCR M range Affy: 1. 66 - 7. 94 Agil: 1. 48 - 6. 17 NCCR: 1. 83 - 7. 12

Agreement: top down 200 (testis) Affy Agilent 41 38 34 46 87 26 53 NCCR M range Affy: -8. 27 - -1. 65 Agil: -6. 07 - -1. 47 NCCR: -6. 18 - -1. 79

Comparison with MPSS, 99% CI (up)

Comparison with MPSS, 99% CI (Down)

MPSS CI Overlap with the 99% CI for MPSS Overlap with the 99. 9% CI for MPSS

Overlap with MPSS 38 74 88 112 MPSS (similar numbers also for Affy and Agilent); 56 of the 88 are in common to all 4 NCCR missing or classified as unreliably mapped (tag to gene not unique)

Conclusions (I) • The three microarray platforms compared performed very similarly in terms of which genes are detected as differentially expressed, distributions of M values, variability between replicate measurements. . . so similarly that it seems hard to find real differences • Most disagreement for low-expressed genes • RMA M values (Affy) are better variance-stabilized, but reproducibility is good for all platforms except for weak signals in Agilent (likely due to bg treatment) • RMA M values are more strongly compressed towards zero at low intensity; reduces false positive calls but might make DE at low intensity undetectable (but is it detectable at all? )

Conclusions (II) Microarrays vs MPSS M values, quantitative comparison: the disagreement is large. . . so large that it is hard to reconcile the values, making it impossible to use MPSS as the ‘gold standard’ M values, qualitative comparison: there is a good degree of agreement - approximately the same to all three microarray platforms

Conclusions (III) • MPSS predicts many more low-abundance genes to be (strongly) differentially expressed • The hybridization methods lose signal of low-abundance genes (due to the background fluorescence estimation? ) • microarrays miss detection of most of the differential expression of low abundance transcripts, but it is also possible that MPSS is biased for many genes or less precise than this approach suggests Þ approach with confidence intervals for MPSS (currently approximate CI that takes into consideration the sampling error on the counts, we have no replicated measurements for MPSS)

Completion of Study Choose genes for q. RT-PCR for which the platforms and MPSS disagree and (attempt to) address the questions: • which platform is more accurate? • • • how does accuracy depend on the signal intensity? do the microarrays miss DE frequently. . ? . . and especially at weak signal intensity ? which platform best detects low abundance RNAs? does MPSS agree with QT-PCR? • Suggestions are welcome !!

Acknowledgements • Ludwig Institute for Cancer Research Victor Jongeneel, Christian Iseli, Brian Stephenson • DAF/DAFL Otto Hagenbuechle, Josiane Wyniger • Uni. GE Robert Lyle, Patrick Descombes • BCF Mauro Delorenzi, Eugenia Migliavacca • and everyone I inadvertently left out!