STAR Grid Activities, OSG and Beyond D. Olsona for the STAR Collaboration The STAR Grid Team: W. Bettsb, L. Didenkob, T. Freemanc, P. Jaklb, L. Hajdub, E. Hjorta, K. Keaheyc, J. Lauretb, D. Olsona, A. Rosea, I. Sakrejdaa, A. Sima a. LBNL, b. BNL, c. ANL STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008
Abstract We will present the ongoing grid efforts of the STAR experiment within the Open Science Grid (OSG) and beyond, as well as the integration of resources in Europe, Asia and South America. STAR is a founding member of the OSG Consortium and has several functioning resources on OSG, its main facilities at BNL/RCF and LBNL/NERSC as well as universities, Wayne & Birmingham. Additional resources are in process of connecting to OSG. Numerous distributed resources used by STAR collaborators are employing grid or grid-inspired technologies. Common examples are the usage of grid job submission tools with the STAR standard workload service called SUMS and the use of data handling and transfer tools across grids. Maximizing on heterogeneity of resources while minimizing in-house platform support efforts, evaluation of the dynamic deployment of reliable data analysis framework via STAR validated software stack with Xen virtual machine is being thoroughly investigated, leveraging advanced VM technologies and research from the CEDPS project. STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 2
Contents • • Background/History Open Science Grid Deployments and Usage Other Distributed Computing Usage Asian Activities Workload Scheduling (SUMS) Virtualization & Cloud Computing Conclusion STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 3
Background/History • STAR has been participating in the U. S. grid activities since the early days of the Particle Physics Data Grid (1999) and a founding member of the Open Science Grid. • Starting with involvement of LBNL and BNL, activities now include collaborators also at Wayne State, MIT, Univ. Chicago, Birmingham, Sao Paolo, Prague and ANL. • Additionally – SUN Grid, 2007 – MIT Xgrid, 2006+ – Xen, Amazon EC 2, 2007+ STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 4
Wayne State University Brookhaven National Lab PDSF Berkeley LAB University of Birmingham Fermi Lab STAR Grid = 90% of Grid resources part of the Open. Science Grid STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008
Amazon. com MIT X-grid Sun. Grid NPI, Czech Republic STAR is also outreaching other grid resources & projects STAR Interoperability / outreach 9 Apr 2008 Virtualization VDT extension Olson, STAR Grid Activities, ISGC 2008 SRM / DPM / EGEE
Resources used by STAR 6 main dedicated sites (STAR software fully installed) • BNL • NERSC/PDSF • WSU (Wayne State University) • BHAM (Birmingham, England) • UIC (University of Illinois, Chicago) Tier 0 Tier 1 Tier 2 Incoming • Prague Tier 2 Other resources • Fermi. Grid - non STAR dedicated ; simulation production 10% level • Sun. Grid – commercial (free for STAR) ; event generation 1 -2% level • MIT Xgrid cluster – analysis mainly ; working on Globus GK for Mac OSX • Amazon. com EC 2 cluster (Elastic Computing Cloud) ; event generation for now ; exercise on Xen based virtualization 1 -2% level STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 7
Be. St. Man SRM Berkeley Storage Manager • SRM interface with caching for data transfer • We use for bulk data transfer as well as asynchronous data placement in job workflow. • Expect to deploy Be. St. Man-Xrootd interface STAR 9 Apr 2008 http: //datagrid. lbl. gov/bestman/ Olson, STAR Grid Activities, ISGC 2008 8
OSG usage Usage - Process Hours / Week STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 9
• Proof of Principle Initial Successes and Benefits from OSG • • Year 1 OSG Milestone for STAR: – Migration of 80% or more of the simulation production to OSG based operation Simulation production - 97% efficiency achieved – Exceeds expectations (we targeted a satisfactory level between 75% to 85% success) Site used are not necessarily STAR dedicated (Fermi. Grid) – Especially: STAR received help from Fermi resources and the FNAL team in June 2007 • several k CPU hours loaned on emergency request • as small as it seems, this help made the difference – This part of resource loan worked and is an important proof of principle of OSG benefit Efficiency of job execution via OSG infrastructure. After resubmission Before resubmission. STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 10
Other grid/distributed activities • Xgrid at MIT – Adam Kocoloski, Michael Miller Leve Hajdu – Mac OS X, 50 desktops – Scavenging spare cycles – Doing STAR data analysis via SUMS so same UI for analysis – Xgrid/Globus job manager in test • Prague, EGEE Tier 2 site – Michal Zerola, Pavl Jakl – High-performance data transfer using multiple srmcp to DPM in Prague (next slide) • SUN Grid – Production of STAR Geant simulations on SUN utility computing resources. STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 11
Data transfer to Prague: parallel srmcp to DPM storage element, 700 Mbps – 20 threads STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 12
STAR Asian institutions • China – – – IHEP, Beijing (2) Institute of Modern Physics, Lanzhou (6) USTC, Beijing (14) Shanghai Institute of Applied Physics (11) Tsinghua University (9) Institute of Particle Physics, Wuhan (12) • India – – – Institute of Physics, Bhubaneswar (4) Indian Institute of Technology, Mumbai (5) University of Jammu (15) Panjab University (5) University of Rajasthan (3) Variable Energy Cyclotron Centre, Kolkata (14) • Korea – Pusan National University (4) – KISTI (in progress as CS collaborator) STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 13
Asian Activities • Many collaborators in Asia • Planning for Tier 2 -like facility at PNU • Discussions with KISTI of possible Tier 1 -like facility for Asia region • Anxious to see how we can better interface/integrate with our Asian collaborators on computational aspects STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 14
Gloriad • 10 Gb all the way through NY • Would allow for immediate full data transfer • Would allow later year ½ dataset transfer – Possibly more depending on Gloriad expansion STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 15
SUMS • STAR Unified Meta Scheduler • A single user interface and framework for submitting to all STAR resources, local and grid flavors • Optimizes resource utilization 25 K jobs/day STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 16
Why Xen? Virtualization? • SIMULATION = EVENT GENERATION IS EASY … – We can all do it … • BEYOND THAT, the reality – Complex experimental application codes • Developed over more than 10 years, by more than 100 scientists, comprises ~2 M lines of C++ and Fortran code – Require complex, customized environments • Rely on the right combination of compiler versions and available libraries • Dynamically load external libraries depending on the task to be performed – Environment validation • To ensure reproducibility and result uniformity across environments • Regression tests cannot be done on all OS flavors due to simple manpower considerations) STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 17
Why Xen? Virtualization? • Solution? Use Virtual Machines (Xen) – – – Bring your environment with you Fast to deploy, enables short-term leasing Excellent enforcement, performance isolation Very good security isolation Minimize experiment team’s efforts • Activity ↔ Development effort leveraged though CEDPS Sci. DAC partner project STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 18
Deploying OSG Cluster as Workspaces OSG CE image as gatekeeper VWS Service Cluster manager can deploy gatekeeper and workernodes in ~ 30 min. Application workload submitted to cluster as to any other OSG CE. Cluster can be retired after workload finishes, freeing resources for other applications. Worker node images with application environment. Pool node Pool node Pool node STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 19
Virtual Machine activities • “Amazon Elastic Compute Cloud (Amazon EC 2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers. ” • Work so far: – Xen image with OSG 0. 6. 0 CE on SL 4. 4 – Xen image with OSG 0. 6. 0 WN on SL 4. 4 – Use Globus Workspaces to deploy gatekeeper and workernodes on EC 2 – Can launch 100 node cluster in ~ 30 min. – Have run Hijing event generator simulations on EC 2. – Have prepared Xen image with full STAR software environment on SL 4. 4, currently being validated • Next steps: – Run event reconstruction of simulations on EC 2 and Teraport cloud STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 20
Nersc PDSF ENC 2 Amazon. com WSU STAR 9 Apr 2008 Accelerated display of a workflow job state Y = job number, X = job state Olson, STAR Grid Activities, ISGC 2008 21
VM image build/maintenance • We are working with r. Path, Inc. in an SBIR project to use r. Builder to efficiently build and maintain OS and application images. • From the inventors of RPM, r. Builder – http: //www. rpath. com/rbuilder – “r. Builder is the first and only development tool that simplifies and automates the creation of software appliances and virtual appliances. r. Builder combines powerful features with innovative packaging techniques to yield a repeatable appliance creation process. “ STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 22
Near term plans • We MUST prepare for real data production on OSG – And take ANY shortcut necessary to accomplish it BY 2009 • onset of DAQ 1000, one order of magnitude higher data acquisition rate than today will require additional resources for real-data processing • Virtualization appears to us as one development helping to easily deploy & run a 2 Million line framework (software) for data mining – UCM job tracking (SBIR with Tech-X) is maturing • Essential to engage discussion on integration – we MUST monitor our application • We have to consolidate our sites – More resources are available in STAR but not-fully used (BHAM, UIC for example) • We will ramp up in infrastructure support to achieve this • We hope leveraging OSG efforts in the US (UIC for example) – We have efforts in integrating Mac OS-X resources from MIT STAR 9 Apr 2008 • Initial work was uniquely started in STAR Olson, STAR Grid Activities, ISGC 2008 23
Longer term needs • Requirements driven by demanding data processing – https: //twiki. grid. iu. edu/twiki/bin/view/User. Group/VOApplications. Require ments#STAR – We will need to efficiently share resources • Concerned about what happens when LHC has ramped up data taking. • Will there be any cycles left to be had? • Additional – STAR is expanding its pool of sites • Interest in sites possibly shared by EGEE - OSG interoperability (especially China) • Hoping for help from OSG to understand policy as well as technology issues. – We believe virtualization is “a” path forward to • Simple deployment of experimental software • Allowing experimental software developer’s team to concentrate on science and a minimal OS version support • Globus workload management needed STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 24
Conclusion • STAR Grid usage is expanding geographically and functionally. • Upgrades at STAR and RHIC are driving a significant increase in computational needs beginning next year which means we MUST push more workload onto the grid. • The emergence (and convergence? ) of VM, cloud computing and grid make very powerful paradigm for scientific computing. • We want (and need) to have greater involvement with our Asia-Pacific colleagues which is enabled with new trans-Pacific networks. STAR 9 Apr 2008 Olson, STAR Grid Activities, ISGC 2008 25
Скачать презентацию STAR Grid Activities OSG and Beyond D Olsona
90c8f4d4204ddda478eb0da9a86e137e.ppt