PRIME 2011 Tutorial PRAGMA Grid Cloud Access Cindy Zheng

PRIME 2011 Tutorial PRAGMA Grid/Cloud Access Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities Pacific Rim Application and Grid Middleware Assembly http: //www. pragma-grid. net http: //goc. pragma-grid. net PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Grid and Cloud • What are they? – Collection of resources for sharing • Differences – Grid • Resources glued by agreed and verifiable trust – Certification – GLOBUS • Provide system/software environment for some applications and users – Cloud • Users setup applications on systems • Cloud provide mechanisms to allow users run their systems as virtual machine (VM) in shared resources PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Grid Trust • Organizations and policies – International Grid Trust Federation (IGTF) • APGrid. PMA • EUGrid. PMA • TAGPMA – Accreditation • Software – Certificate Authorities (CA) • Set up CA services, policies • Issue/manage certificates (user, services, …) – GLOBUS • Identify users and resources in a cohesive fashion • Map user certificate to a local account • Interface local job managers PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

PRAGMA CAs • PRAGMA-UCSD CA (https: //goc. pragma-grid. net/ca) – Accredited by APGrid PMA – Included in IGTF distribution – Only issue a certificate when needed • PRAGMA Experimental CA (http: //rocks-56. sdsc. edu/exp-ca) – Only accepted by PRAGMA grid site • PRIME 2011 students will use PRAGMA experimental CA unless your project involved with other grids PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Grids/Clouds and PRAGMA Grid/Cloud • Grids/Clouds – Most are centrally funded • PRAGMA grid/Cloud is unique – Grass-root • Voluntary contribution • Open (PRAGMA member or not, pacific rim or not) • Long-term collaborative working experiment – Heterogeneous • • Funding No uniform infrastructure management Open to all sciences and applications All varieties of site policies, system and network environments – Friendly, helpful, close relationships – More challenging and rewarding • Good for development, collaborations, integrations and testing PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

PRAGMA Grid/Cloud UZH Switzerland CNIC China JLU China LZU China Uo. Hyd India HKU Hong. Kong KISTI KMU Korea ASGC NCHC Taiwan SDSC USA HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam IHPC/NGO NTU Singapore MU Australia Indiana. U USA CICESE UNAM Mexico ASTI Philippines NECTEC KU Thailand MIMOS USM Malaysia AIST Osaka. U UTsukuba Japan BESTGrid New Zealand Ce. NAT-ITCR Costa Rica UValle Columbia UChile 26 institutions in 16 countries/regions, 22 compute sites, 9 VM sites (+ 6 site in preparation)

PRAGMA Grid Software Layers http: //goc. pragma-grid. net/pragma-doc/userguide/join. html Applications FMO Savannah MM 5 CSTFT Siesta AMBER Phylogenetic … Ninf-G Nimrod/G Mpich-GX … Gfarm SCMSWeb CSF MOGAS … Application Middleware Infrastructure Middleware Globus (required) SGE PBS LSF SQMS … Local job scheduler (require one) PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

PRAGMA Grid/Cloud Compute Resources http: //goc. pragma-grid. net/pragma-doc/computegrid. html PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

PRAGMA Grid/Cloud Members and Team http: //goc. pragma-grid. net/wiki/index. php/Site_status_and_tasks • Team members – – – >240 and growing one management contact / site 1~3 technical support contact / site 1~4 application drivers / application 1~5 members / Middleware development team – – Application (various scientific domains) Middleware (various grid middleware) Security (grid security) … • Experts • Communications – Email – Mailing list – pragma-grid-team@googlegroups. com – Coordinator – zhengc@sdsc. edu PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

PRAGMA Cloud Access • Cloud computing is new – Access procedure has not been formalized • Contact Cindy for detail info and arrangements • The rest of the slides shows how to access PRAGMA grid PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Obtain PRAGMA Grid Access • Find guide document at http: //goc. pragmagrid. net/wiki/images/5/51/PRIME-accounts. doc • Be aware – When cut/paste a command line • Microsoft word may change – A hyphen to a line character – Double-hyphen to a long line character • Learn UNIX concepts and commands Ø You need for your project Ø Know how Ø Understanding (Know why) Ø Account application as an example Ø Learn more online, google http: //mally. stanford. edu/~sr/computing/basic-unix. html Ø Try them, hands-on Ø But be careful with “rm” (remove) command! PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Same Basic UNIX Commands • List files $ ls -l (show file permission, ownership, etc. ) $ ls -a (show all files, including hidden files) • Change directory $ cd (go to my home directory) $ cd ~/. ssh (go to. ssh subdirectory under my home directory) • Copy files locally $ cp • Copy files between local and remote systems use SSH key $ scp : $ scp : • Remember the command, but forgot how to use it $ --help $ man PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Need an UNIX System on Laptop (Step 1. 1) • UNIX on laptop – MAC OS 10 ØHD ØApplications ØUtilities ØTerminal – Windows ØInstall Cygwin ØClick – LINUX ØLogin PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Create SSH Key (Step 1. 2) • Secure remote login • Generate key files $ ssh-keygen -t rsa Ø Use default path for ssh key files Ø Good passphrase (hard to guess, no dictionary words) Ø Remember the passphrase • View SSH key files (private vs. public) Ø From Cygwin $ ls ~/. ssh -r----@ 1 cindyzheng staff 951 Mar 19 01: 10 id_rsa -rw-r--r--@ 1 cindyzheng staff 234 Mar 19 01: 10 id_rsa. pub -rw-r--r--@ 1 cindyzheng staff 8414 Mar 19 01: 10 known_hosts Ø From Windows c: Program filesCygwinHome. ssh • Backup SSH key files • • USB drive, take with you Can be used thereafter Only on secure systems Possibly compromised key • Immediately inform all public key holders (system administrators) PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Account Application (Step 2) • Email to Cindy – – – – Your full name Institution name: UCSD Address of the institution: UCSD Country of the institution: USA Your phone number Your fax number Your email address The purpose of using PRAGMA grid: PRIME Your/your institution's contribution to PRAGMA (can be work, resources): work Prefered login names (1 st, 2 nd, 3 rd, but may or may not be granted) Prefered UNIX shell (may or may not be granted) Hostname(s) and IP address(es) of the system(s) which you launch your applications (This is required for PRAGMA grid site systems to open their firewalls in order to allow you access. ) Your ssh PUBLIC key (NEVER and NEVER give anyone your private key file!! Nor your ssh password!!) PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Cindy • Create user info pack • Create an UNIX user account on rocks-200. sdsc. edu • Email user – Account name on rocks-200. sdsc. edu – Certificate request license id PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Test SSH key (Step 3) • Login to rocks-200. sdsc. edu $ ssh @rocks-200. sdsc. edu Enter passphrase Last login: Tue Apr 5 11: 37: 56 2011 from dyn 137 -110 -115232. ucsd. edu Rocks 5. 3 (Rolled Tacos) Profile built 04: 25 15 -Apr-2010 Kickstarted 22: 21 14 -Apr-2010 • In case ssh fails $ ssh –v @rocks-200. sdsc. edu Email output to Cindy First time login on rocks system, ssh-keygen ran automatically Accept default ssh file path Empty password • Only used internally • not recommended elsewhere PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Make a User Certificate Request (Step 3) $ cd $ /opt/pragma-ca/bin/grid-certreq -sv ra. pragma-grid. net: pragma-exp_ra -new < license_id> -g "SMIME user" -sou -----(Sample output)------creating a certificate signing request ---------------------generate private key (size 1024 bit). . . . oo. . . . . oo ------- input user subject information -------email can be omitted by putting a char of '. ’ input ou : PRIME type exactly as this input user name : Cindy Zheng input your full name input user email : cindy@sdsc. edu input your email address, must! ------- please confirm your inputs ------GROUP : SMIME user SUBJECT : CN=Cindy Zheng, Email=cindy@sdsc. edu do you continue operation? (yes/no/retry)[y]: press return trying to connect RA server : ra. pragma-grid. net (11412). . . ok. request for issuing a new certificate. . . ok. your request is accepted. (Accept. ID=0000003) <- remember it CA operator will send an email to tell a result. save a CA certificate file : /home/cindy/. globus/cacert. pem <- note cert request files location save a private key file : /home/cindy/. globus/userkey. pem <- note cert request files location Input PASS Phrase: type a good password Verifying - Input PASS Phrase: retype the password (remember it!) • Email Cindy the accept. ID PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Cindy • Issue user certificate – Off-line CA server • Inform user via email PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Retrieve User Certificate (Step 4) • Login to rocks-200. sdsc. edu $ ssh @rocks-200. sdsc. edu When prompt, enter your SSH password • Retrieve user certificate $ /opt/pragma-ca/bin/grid-certreq -sv ra. pragmagrid. net: pragma-exp_ra -em -recv trying to connect RA server : ra. pragma-grid. net (11412) request for exporting a certificate. . . ok save a CA certificate file : /home/cindy/. globus/cacert. pem save a certificate file : /home/cindy/. globus/usercert. pem • View and understand certificate files $ ls -l. globus -rw-r--r-- 1 cindyzheng staff 2878 Mar 23 18: 26 cacert. pem -rw-r--r-- 1 cindyzheng staff 4998 Mar 23 18: 26 usercert. pem -r---- 1 cindyzheng staff 1743 Mar 23 18: 26 userkey. pem PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Backup Certificate Files (Step 4) • Copy. globus directory from rocks-200. sdsc. edu to your laptop Ø Start Cygwin/terminal on your laptop (Should land in your home directory) Ø $ scp -r @rocks 200. sdsc. edu: /export/home//. globus. • Copy c: Program filesCygwinHome. globus to your USB backup disk • • Take with you anywhere Can be used usually 1 year Only on secure systems Possibly compromised certificate • Immediately inform certificate issuer (Cindy) PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Test Globus Authentication (Step 5) On rocks-200. sdsc. edu: • Create a proxy $ grid-proxy-init Your identity: /O=grid/O=pragma/OU=PRIME/CN=Cindy Zheng Enter GRID pass phrase for this identity: <- input passphrase Creating proxy. . . . Done Your proxy is valid until: Thu Apr 7 02: 18: 11 2011 • Test globus authentication $ globusrun -a -r rocks-200. sdsc. edu GRAM Authentication test successful • Email Cindy if successful, otherwise email output to Cindy • To create longer proxy, specify hours $ grid-proxy-init –hours 100 PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Test Job Submission (Step 5) $ globus-job-run rocks-200. sdsc. edu/jobmanager-sge /bin/date /opt/gridengine/default/spool/rocks-65/active_jobs/330008. 1/pe_hostfile compute-0 -12 Tue Apr 21 13: 06: 18 PDT 2009 rm: cannot remove `/tmp/330008. 1. all. q/rsh': No such file or directory ignore this for now Note: depending on how busy the system is, you may need to wait a bit. To find out the status of SGE queue, open/login to another shell and run $ qstat –f queuename qtype used/tot. load_avg arch states --------------------------------------… --------------------------------------all. q@compute-0 -6. local BIP 1/2 0. 06 lx 26 -x 86 328085 0. 56000 data jas dr 04/15/2009 23: 29: 01 1 --------------------------------------… ################################### - PENDING JOBS ################################### 330010 0. 00000 data zhengc qw 04/21/2009 13: 18: 39 1 Email Cindy if successful, otherwise email output to Cindy PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Cindy • Add your email to pragma-grid-team list • Email pragma-gridteam@googlegroups. com ask site adminitrators to create account on their systems • Site administrator will email your username to you and me after your account is created PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Test Access To Your Host Site (Step 6) • Host system name/ip (where you will be launching your applications from) • Do the same tests as you have done with rocks-200. sdsc. edu – Test SSH (Slide 20) – Test Globus authentication (Slide 29) – Test job submission (Slide 30) • More testing tips o http: //goc. pragmagrid. net/wiki/index. php/User_Testing o Google PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Test Access to Each Site (Step 6) • Test SSH – Find the target system name at http: //goc. pragma-grid. net/wiki/index. php/Site_status_and_tasks – Test SSH from your host system to the target host (should work) – Test SSH from your laptop to the remote host (May work) – Issues about firewall and site policies • Test Globus authentication – From the system at your host site – To the target system • Test job submission – From the system at your host site – To the target system • Problem with any test – Save output – Check tips at • http: //goc. pragma-grid. net/wiki/index. php/User_Testing • Google – Email site contact and cc zhengc@sdsc. edu PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011

Process At A Glance PRIME Student start Laptop platform MAC Cindy Step 1. 2 Create a SSH key w i n d o w s Step 1. 1 Install Cygwin Step 2 Email application Create account on rocks-200 Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission Step 6 Test SSH, Globus and job submission to each site PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011 confirmation sites reply Request to all sites

Use PRAGMA Grid • Information online goc. pragma-grid. net Ø member sites (hostname, contacts, user guide, …) Ø computational grid (resource table, jobmanager, cpu architecture, …) Ø Applications (as examples) Ø SCMSWeb (realtime system status, software catalog, etc. ) wiki. pragma-grid. net Ø For Users • Ask questions (email, skype) Ø Ø Email, skype Site contact – always cc Cindy Looking for experts – ask Cindy If Cindy is not around, pragma-grid-team@googlegroups. com PRIME Tutorial 4/22/2009 Prime Tutorial, 4/22/2011