Hardware Commissioning Review Markus Zerlauth 11 th July

Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 CO deliverables for HWC – lessons learnt and the proposed way ahead Markus Zerlauth, AB-CO-MI for the CO group 1

Outline è Post Mortem analysis and test automation Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 è Circuit Synoptic è Logging è CCC environment for HWC è E-logbook è Conclusion 2

Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 Post Mortem Analysis 3

Current PMA packages 3. PNO. 2 auto analysis Q 5 D 2 Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 Q 4 1 s Courtesy of PM team 4

Current Architecture of PMA system v 5 Raw data Analysis sequencer Individual system analysis Result data 1 Logical system analysis Result data 2 General system analysis PC Circuit 600 A PIC 1, 2 PIC ready 29 -06 -2007 Circuit Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 60, 80, 120 A QPS Electrical circuits Circuit SDDS Analysis sequencer Circuit logging GSA Magnet system Circuit Cryo circuits Equip Vac Equip BLM Courtesy of A. Rijllart DFB, SCLink Stand. cell additional results 5

Post Mortem Analysis – further automation è No fully automatic analysis feasible yet, but further automated hand-shake between sequencer and PMA possible è Complete Logical system analysis + with further dedicated analysis packages of type PNO. 2 Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 è Priority must be on circuit types <6 k. A (largest quantity) è PO, PMA and sequencer teams are investigating possibilities to further automate PO related tests for 60 A and 600 A circuits (PCC, crow-bar, etc. . ) è Issues: – In-homogeneous test plan & interest of involved parties requires global coordination of priorities for • Final sequences (need to agree on last changes & freeze as much as possible) • Related automated analysis – Development priorities need to be urgently agreed upon in between all involved parties in the next weeks to provide operational & tested packages already for sector 45 – Automated test plan versus manual (MTF reporting, parameter update, . . ) è Series of SACEC meetings to address these issues 6

Sequences versus ‘automated’ PMA 600 A IPD IPQ 13 k. A x 1036 X 410 X 16 X 78 X 32 PIC 1. x PCC PCC PCC PIC 2. x Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 60 -120 A PIC 2. x PNO. 1 PCS. 1 -4 PLI 1. x PCS. 1 -4 PNO. 2 PLI 1. x PLI 2. x PLI 1. x PNO. 3 PLI 2. x PLI 3. x PLI 2. x PNO. 4 PLI 3. x PNO. 5 PLI 3. x PAC PNO. 1 PNO. 6 PNO. 10 PNO. 7 PNO. 6 PNO. 10 PAC PNO. 10 PSQ PAC PM analysis automated Sequences ready PAC 7

Post Mortem Analysis – ongoing improvements è PM Data Browser – Automated loading of correlated PM data from QPS/PIC/EE, etc. based on the FGC event Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 – Event search based on circuit name and/or time window – possibility to manage the list of the signals to display in the HWC view (via a text editor) – user can save the jpeg of the CHART or the excel sheet from the TABLE in a local folder – Link with cryogenic signals • link between Electrical circuits & cryogenic equipment foreseen to be established in the Layout DB • used for PM Data browser, Cryo SCADA, CIET and Circuit Synoptics • scripts generating a set of signals / circuit 8

Post Mortem System – scalability è Dedicated scalability tests + HWC experience have not shown any bottlenecks in the available PM infrastructure (data collection on PM server and data conversion) – in case it does in the future, further servers will be deployed Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 è Current limitations mostly lie with the gathering of data on the client side è PM scalability @ example of AUG test 09 th July at 9: 50: 28 – 49 ext FGC events, 77 self FGC events, 17 DQAMS events – The PM data was received on the server in two batches, each ~1 min long: at 9: 51 and 9: 56. No errors on the server side, no client disconnections detected during the test. – Conversion time to pmd format ~50 ms for 600 A converter file, ~500 ms for 60 A converter file, ~40 ms for DQAMS file 9

Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 Circuit Synoptic 10

Circuit Synoptic è Circuit Synoptic an important tool during HWC, gathering necessary information related to circuit powering on a single screen è 2 major improvements proposed for next sectors Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 – Inclusion of 60 A converters (60 A SW power permit and FGC status) – Link of circuit synoptics with related cryogenic signals, e. g. CL temperatures, liquid levels in related DFBs, CRYO_START, CRYO_MAINTAIN, etc. . . è Requires considerable work on configuration DB for mapping of signals (together with ACR) + data publishing from CRYO system (ACR & CO), but could be exploited by circuit synoptic, CIET, CRYO SCADA, etc. . è Would allow for ‚real-time‘ trends of e. g. circuit current vs. current lead temperature within PVSS, etc. . . Courtesy of F. Bernard 11

Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 Circuit Synoptic v 2 Courtesy of F. Bernard 12

Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 Logging 13

The LHC Logging Services - Architecture 60, 000 records / day LHC Logging Service JDBC Measurement DB 10 g AS Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 JDBC PL/SQL HTTP 6, 800, 000 records / day JDBC Generic Usage Logging DB 147, 000 records / day PL/SQL HTTP 129, 000 records / day Technical Services DB 14 Courtesy of C. Roderick

Logging - Infrastructure + Interface Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 è HW infrastructure – current Logging service has to cope with rapidly increasing demands – initially intended to deal purely with LHC machine/beam related data; yet today, data comes from PS Complex, SPS-EA, CNGS, ATLAS, LHCb, CMS, CIET… – Worry about interdependency with LSA (currently on the same database server) + scalability for 8 parallel sectors – DM has launched a study and will present a proposal for the backend upgrades at a TC on August 16. HW upgrades are tough to be in place for October 2007. . . è Data extraction interface – technology behind the current data extraction tool allows for an easy webaccess from inside and outside CERN, but is for this reason also specific and by now outdated – Already planned to be replaced new implementation, thus addressing the open issues such as compatibility, performance and flexibility – implementation has partially started but the increasing workload for projects such as LSA, hardware commissioning, . . has made it impossible to further progress up to now (estimated to 2 man-months, due to specific knowledge no easy 'outsourcing‘ possible) – Currently on hold…. 15

Logging & Data providers – example of CRYO data è The present Logging system is used by most players of HWC, which store their system data at rates and accuracies which are primarily defined by system experts and inherent limitations of their systems (while CO will discuss with every client to Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 minimise data volumes for long-term storage if possible) è For the global event analysis, additional data of in this case the cryogenic system has been retrieved, but first attempts showed limited time resolution and accuracy è First meeting with experts revealed several possibilities of improvement in the controls infrastructure that deserve follow-up, but also inherent limitations of the acquisition electronics -> see also summery paper of Mike (to be released) è Requirements for data storage of system designers (doing process control & system setup) and HWC & MPP experts (doing event analysis) can be very different and should be discussed by the latter two in case the provided accuracy and data rates are not sufficient for HWC activities 16

Logging – CRYO data The CRYO SCADA system Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 LT data FEC PLC Time WFIP Signal 1 Hz stamp 0. 1 Hz 5 Hz Processing (median calculation) SCADA Archive SCADA Threshold (0. 08%) Threshold (0. 1%) db Dead time (10 s) The CIET system LT data 0. 1 Hz FEC WFIP 5 Hz Courtesy of M. Koratzinos Signal Processing (median calculation) Delay 30 s SCADA db Threshold (0. 5%) 17

Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 CCC Environment + elogbook 18

Comments on working environment… è Generally, CCC environment very much appreciated by teams è Passwords, account-timeouts, CMF updates and limited access to GPN perceived as disturbing during HWC, but need to follow new CERN security policy Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 è Unique way to start programs from Linux and Windows consoles not easily possible as many clients only run on Windows (PVSS) or Linux, but to be followed up è Printing from the different consoles perceived difficult è Installation of Lab. View 8. 2 on Windows consoles to run PM application directly è Elogbook – Vital tool for tracking and re-construction of HWC activities (EICs can confirm, especially when multiple tests are ongoing and time is limited) – Operators have the good habit to accurately fill these logbooks – Especially for parallel commissioning, changing shifts and if tests are carried out manually (no LSA tracking) an asset è Do we need further μCCC during HWC? 19

Conclusions and Outlook è Many CO services involved during sector 78 commissioning, which all gathered valuable input for further evolvement and improvements that will be digested and implemented during the next weeks Hardware Commissioning Review, Markus Zerlauth, 11 th July 2007 (timing, infrastructure, PIC, WIC, CRYO SCADA, CIET, QPS SCADA, PMA, applications, World. FIP, databases, Logging, etc. . . ) è Further input needed from HWC and MPP to have asap a common view on priorities for further improvements, test automation & global analysis -> SACEC è Issues as databases, timing, etc not treated here but will be adressed with concerned teams -> SACEC è Feedback from equipment groups & this HWC review will be addressed and followed up within the CO group during a dedicated technical commitee next week 20