Science Engineering Technology and the Facilities that Support

Science, Engineering, Technology… (and the Facilities that Support them) San Diego Supercomputer Center University of California, San Diego Net@EDU Annual Meeting February 5, 2007 Dallas Thornton IT Director, SDSC SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego

SDSC in a nutshell Grid and Cluster Computing n n n Networking n Employs nearly 400 researchers, staff and students UCSD Organized Research Unit Strategic Focus on Data-Oriented Scientific Computing Home of many associated activities including o o Integrated Biosciences o o o Geosciences Network (GEON) Network for Earthquake Engineering Simulation IT (NEESit) Protein Data Bank (PDB) Joint Center for Structural Genomics Alliance for Cell Signaling (Af. CS) Biomedical Informatics Research Network (BIRN) Coordinating Center High Performance Wireless Research and Education Network (HPWREN) SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego High-end computing Data and Knowledge Systems Integrated Computational Sciences

A Partial List of Databases and Data Collections currently housed at SDSC n n n n n n Protein Data Bank (protein data) National Virtual Observatory (astronomical data) UCSD Libraries Image Collegion (Art. Store) National Science Digital Library (education collection) SCEC (earthquake data) BIRN (neuroscience data) Encyclopedia of Life (genomic data) Protein Kinase Resource (protein data) Tree. Base (phylogeny and ontology information) Transport Classification Database (protein information) Plants. P (plant kinase information) Plants. T (plant transporter information) Plants. UBQ (plant protein information) CKAAPS (protein evolutionary information) Af. CS Molecule Pages (protein information) SLACC-JCSG (structural genomics data) APOPTOSIS DB (proteins related to cell death data) NAVDAT (geochemistry data) QRC (NSF data on Supercomputer Centers and PACI) Network Topology Data (Skitter project) Biology Workbench Databases (mirrors and “originals” of over 80 biology databases) San Diego and Tijuana Watersheds (water resources mapping) • • • • • • 2 Micron All Sky Survey (astronomy data) Digital Palomar Observatory Sky Survey Collection (astronomy data) Sloan Digital Sky Survey Collection (astronomy data) Interpro Mirror (protein data) HPWREN Wireless Network Analysis Data HPWREN Sensor Network Data Security logs and archives (security information) Nobel Foundation Mirror (information) Earth. Ref Digital Archive (Earth Science information) GERM (earth reservoir information) PMAG (paleomagnetic information) GEOROC (petrological and geochemical data for igneous rocks) Kd’s DB (rocks and minerals) Braindata (Rutgers neuroscience collection) LTER (hyperspectral images) SIO-Explorer (oceanographic voyages) Scripps (oceanographic research data) Transana (classroom video) Web. Base (web crawls) Alexandria Digital Library (photographs) Backskatter Data (from UCSD network telescope) Digital Earth Data Library (earth sciences related datasets) SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego • • • • • PETDB (petrological and chemical data) Seamount Catalogue (bathymetric seamount maps) IPBIR (primate information) Hayden Planetarium Collection (astronomical data) Tera. Grid Data (science and engineering collections) Digital Embryo (human embryology) National Archives (persistent archive) San Diego Conservation Resources Network (sensitive species map server) Bionome (Biology network of modeling efforts) KNB (Knowledge networks for biocomplexity) LDAS (land data assimilation system) SEEK (ecology data) ROADNET (sensor data) NPACI Data Grid (scientific simulation output) Salk (biology data archive) CUAHSI (community hydrological collection) Backbone Packet Header Traces (OC 48, OC 12)

SDSC’s Funding n n n Federal Grants State Support Campus Support Industry Partnerships Recharge / Fee For Service Leverage Economies of Scale o Labor – Consulting, Support, Sys Management, etc. o Storage o Compute Cycles o Collocation/Hosting Services o SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego

SDSC’s Evolutionary Datacenter n n n Privately-built 7, 000 sq ft. in 1985 Transitioned to UCSD in 1997 Expanded to 11, 000 sq. ft. in 2001 Expanded to 14, 000 sq. ft. in 2006 Expanding to 19, 000 sq. ft. in 2008 Power and Cooling Requirements Grew and Changed with New Systems Previous upgrades have been costly. o Developing a scalable power and cooling infrastructure with UCSD facilities to accommodate future systems. o SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego

Lessons Learned n Maximize yield from the build and upgrades o o n Incremental upgrades are exceedingly expensive! Engineer the facility for 2 x-4 x power, cooling, and space expansion capability. . . (No matter what the architects say. ) Decide where to invest your money o 2 N configurations, UPSes, Generators, etc. are great but usually too expensive to be worthwhile for large research clusters. n n Evaluate systems in need of this reliability and build accordingly. Consider different rates for this extra level of service. Be on the same page with campus facilities o n (or Learning) Ensure newly-installed distribution paths provide spare capacity. Carefully evaluate utilities costs in site selection. Standardize, standardize! SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego

Q&A SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego

The Density Problem Note Log Scale HPC Even More Dense 10 k. W Racks in 2005 will be 100 k. W in 2010 Rising Density + Reduced Costs = Exponential Demand Growth SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego

Who pays for the facilities? n PIs / Faculty o What do my indirect costs pay for, anyways? n n n Grantors o Facilities should be funded by the state. n n n This varies widely by institution, but IDCs do not scale well with the facilities requirements of machines over time. Need to budget incremental facilities costs in grants. As the costs to operate and maintain increasingly facilities-hungry systems increase, states are less capable of providing adequate support. Need to support incremental facilities costs in grants. Campuses/States o The grantor should pay the costs of the grant’s needs. n n A valid argument, but if the state/campus wants to be competitive with their proposal, some subsidy is required. Need to develop a scalable model to incrementally fund facilities, decide how much this will be subsidized, and get buy-in from PIs and Faculty. SAN DIEGO SUPERCOMPUTER CENTER at the University of California, San Diego