Computing at Fermilab David J Ritchie Computing Division

Computing at Fermilab David J. Ritchie Computing Division May 2, 2006 May 2006 David J. Ritchie: CD-doc-800 -v 1

Computing At Fermilab is Essential To: • • Control the Accelerator. Record the Detector information from the Collisions. Reconstruct the Particles’ Parameters. Globally Distribute the Data and Calibrations. Physics results from the Particles. Estimate the Errors. Publish our Discoveries. Simulate everything to check the answers. May 2006 Discuss and Analyze across Physicists Worldwide David J. Ritchie: CD-doc-800 -v 1 2

Computing At Fermilab is At the Edge • Respond to Accelerator time scales of Pico. Seconds (10**-12) – Special-purpose Custom Built “Processors” with DSPs etc • Collect data at Gigahertz (10**9 words/second). – Use specially built parallel optic fibers from DSPs to 100 MByte buffers. • Filter information in real time – ~150 PCs operating in parallel to read and compute on the data in real time before. • Record data to tape at 20 MBytes/sec all the time the accelerator is working. … and … May 2006 David J. Ritchie: CD-doc-800 -v 1 3

Computer Security The Dark Side of the Force • The Internet is now ~100 Million computers/users: “one in a million” events happen every day. • With the net ‘somewhere on the net’ is always just next door. • We are learning how to live in a world where not everyone is friendly -- but most are. • A recent demonstration of an unpatched Mac showed it was hacked in 2 minutes. • We use automated scanning software to detect problems and always follow up on any inkling of a problem. May 2006 David J. Ritchie: CD-doc-800 -v 1 4

Accelerator Controls: Measuring where the Beam is May 2006 David J. Ritchie: CD-doc-800 -v 1 5

How Do You Do Science? (elementary school version) 1. 2. 3. 4. 5. 6. 7. Observe phenomena. Develop a hypothesis. Use hypothesis to make predictions. Devise experiment to look for predictions. Obtain results demonstrating (or not) predictions. Draw conclusions about correctness of hypothesis. If correct, add to accumulated hypotheses which are theory. If not, revise hypothesis and … Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 6

How Do You Do Science? (Large Scale Science Version) • • • Each stage of scientific process has different computing needs. Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan and Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 7

Computing Needs Phenomena and Hypothesis Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Need: access previous results • PC’s (Windows, Linux), UNIX, • • Macs Printers – color and B/W World-wide networking MSWord, Te. X, Acrobat Web preprint repositories – Arxiv. org at Cornell U. – Spires at Stanford U. Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 8

Computing Needs Organize Collaboration Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Need: • PC’s Macs (Windows, Linux), UNIX, • World-wide networking • E-mail • Web Pages – DZero “Top Group” – MINOS Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 9

Computing Needs Propose Experiment Each stage of scientific process has different computing needs. • • • Need: with colleagues show Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions the feasibility of a proposal. • PC’s Macs (Windows, Linux), UNIX, • World-wide networking • Web pages – Dark Energy Camera Project – Nova Proposal • Simulations… Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 10

Simulations are done throughout the experiment lifetime -before, during, after data taking. e. g. CDF Silicon Vertex Detector A 3 D modeling program simulating particles in the detector. May 2006 David J. Ritchie: CD-doc-800 -v 1 11

Computing Needs Get Approved and Funded Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Need: Project Management Funds come from DOE, NSF, NASA, Congress, Taxpayer—YOU. • PC’s (Windows, Linux, Macs) • Printers – color and B/W • World-wide networking • MSProject, Power Point… • Proposal review… –NUMI Proposal Review Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 12

Computing Needs Plan / Design, Build / Install Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Need: simulation, sensor design, engineering software…, • Unusual sensors • Custom electronics • Computing at its core: microprocessors, silicon logic Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 13

Unusual Sensors May 2006 David J. Ritchie: CD-doc-800 -v 1 14

Computing Needs Acquire / Record / Store Data Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Repeat May 2006 Need: Acquire, record, store • Use sensors and custom electronics David J. Ritchie: CD-doc-800 -v 1 15

Acquire, Record, Store Data May 2006 David J. Ritchie: CD-doc-800 -v 1 16

Recording Data • Record data (“events”) – The sensors and custom electronics convert the voltages and currents into 300, 000 bytes of 1’s and 0’s: Now, we are in the computer realm. – Get an event every few hundred microseconds. – Raw data rate — hundred’s of GB/second. • Either trigger on interesting events or filter out the uninteresting ones – ~10 MB/s. • What gets through, one records to disk and eventually to tape. May 2006 David J. Ritchie: CD-doc-800 -v 1 17

Computing Needs Analyze Data Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Acquire / Record / Store Data Analyze Data Obtain Results Publish Conclusions Need: • Reconstruct 1’s and 0’s into tracks and particle identification • Do physics analysis on resulting collections of tracks Means: • PC farms, lots and lots • Disk backed up by tape with robotic tape mounts Enstore Usage. Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 18

100 Programmers to make 1, 000 lines of code to get to the results (ack. L Sexton-Kennedy) May 2006 David J. Ritchie: CD-doc-800 -v 1 19

Computing Needs Obtain Results Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Need: Do additional physics analysis. Means: Use analysis farms with their disk caching capabilities to speed data analysis. CDF Usage Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 20

ROOT May 2006 An object oriented HEP analysis framework. David J. Ritchie: CD-doc-800 -v 1

A Framework provides utilities and services. CDF Tops the Top World Average Results Are often histograms May 2006 David J. Ritchie: CD-doc-800 -v 1 22

And in Astro. Physics May 2006 David J. Ritchie: CD-doc-800 -v 1 23

To Public Data Release May 2006 David J. Ritchie: CD-doc-800 -v 1 24

Computing Needs Publish Conclusions Each stage of scientific process has different computing needs. • • • Identify Phenomena Develop Hypothesis Organize Collaboration Propose Experiment Get Approved Obtain Funding Plan / Design Experiment Build / Install Equipment Record / Store Data Analyze Data Obtain Results Publish Conclusions Need: Show evidence of progress in scientific work. Means: Publish papers and conferences. Top Quark Mass Repeat May 2006 David J. Ritchie: CD-doc-800 -v 1 25

Mass of the Top Quark May 2006 David J. Ritchie: CD-doc-800 -v 1 26

Vital Statistics • • • FNAL has >10, 000 network connected devices They are connected by >1, 000 miles of cabling There is more than 200 TB of disk spinning now We have over 4 PB of data on tape Half of it is less than 2 years old May 2006 David J. Ritchie: CD-doc-800 -v 1 27

Moore’s Law Lots of Data! Data Volume doubles every 2. 4 years May 2006 David J. Ritchie: CD-doc-800 -v 1 28

4 Petabytes of Data stored in the Fermilab Robots 4 Petabytes = 4, 000 TB = 4, 000 Gigabytes or about 100, 000 Disks on your PC. Data In Fermilab Robots Equivalent to a stack of CDs nearly 10 times as high as the Eiffel Tower May 2006 David J. Ritchie: CD-doc-800 -v 1 29

Computing Facilities • The activities map into facilities at the laboratory: – Networks – Mass Storage robotics / tape drives – Large computing farms – Databases – Operations – Support May 2006 David J. Ritchie: CD-doc-800 -v 1 30

D 0 computing systems “central-analysis” “data-logger” “d 0 -test” and “sam-cluster” 12 -20 MBps 100+ MBps 12 -20 MBps Enstore Mass Storage System “farm” “linux-analysis -clusters” “linux-build-cluster” May 2006 400+ MBps David J. Ritchie: CD-doc-800 -v 1 “clue. D 0” ~100 desktops 31

No Data Center is an Island Run II -- D 0 planned for Regional Analysis Centers (RAC’s) since 1998: – Distributed analysis around the globe UO UA LTU UTA FSU Rice CINVESTAV (arbitrary imagined distribution) May 2006 David J. Ritchie: CD-doc-800 -v 1 32

And for the LHC in 2007 the CMS Experiment - example of a global community grid CERN Taiwan Italy UK USA@FNAL Germany France UNL Purdue Wisconsin Caltech UCSD Florida MIT Data & jobs moving locally, regionally & globally within CMS grid. Transparently across grid boundaries from campus to the world. May 2006 David J. Ritchie: CD-doc-800 -v 1 33

US CMS Computing • Fermilab is the host lab of U. S. CMS experiment which will begin taking data at CERN in Geneva, Switzerland in ~2007 • Fermilab hosts the project management for the U. S. CMS Software and Computing Program in DOE • U. S. Physicists will participate in this research. May 2006 David J. Ritchie: CD-doc-800 -v 1 34

CMS Data Xfer FNAL to World The US CMS center at FNAL transfers data to 39 sites worldwide in CMS global Xfer challenge. Peak Xfer rates of ~5 Gbps are reached. May 2006 David J. Ritchie: CD-doc-800 -v 1 35

~45 Sites Working together as a Coherent Computing Facility 2500 jobs May 2006 ~20 Research Groups Sharing the Sites to Run Compute Intensive Jobes David J. Ritchie: CD-doc-800 -v 1 36

Collaborations across Continents and across Disciplines coarse-grained particle ion channel simulator based on the Boltzmann Transport Monte Carlo methodology. May 2006 David J. Ritchie: CD-doc-800 -v 1 37

The Grid is for Everyone. How will it be used? May 2006 David J. Ritchie: CD-doc-800 -v 1 38

Acknowledgements As always in a laboratory, one gains from the work of one’s colleagues. This talk is no exception. I would especially like to complain about Ruth Pordes who was meant to give this talk!. May 2006 David J. Ritchie: CD-doc-800 -v 1 39