LHC-OPN operations Roberto Sabatino LHC T 0 T 1

LHC-OPN operations Roberto Sabatino LHC T 0/T 1 networking meeting Rome, 4 April 2006

From 31 January • Integrate LHCnet and Taiwan • Relationship with EGEE-SA 2 / ENOC use cases • Progress of GN 2 JRA 4 and perf. SONAR

LHCNET+TAIWAN • LHCNET – Based on 10 Gbps circuits between Force 10 switches – Possible and willing to extract up/down status – Hence is possible to integrate in operational monitoring model • TAIWAN – pending

ENOC use cases • Document sent to list, resulting from discussions between EGEE-SA 2, DANTE, DFN • 2 functions (data model, weathermap) and 5 workflows described

WF 1 -T 1 notices a fault in connectivity to T 0 • • the networking staff at T 1 will inform the GGUS via a TT T 1 (belongs to NREN A) may contact the NREN A following the national procedures: – It is a known fault (or scheduled maintenance) in which case GGUS will reissue the trouble ticket, or point to the repository of open tickets. – If it is a new fault, workflow 2 below applies

WF-2: The ENOC takes TT from GGUS/gives feedback to GGUS • ENOC analyses where the fault is. Two cases apply: – – • The ENOC via the weathermap is able to automatically locate the fault. The ENOC will signal the fault to the NREN involved via a trouble ticket and await further information from that NREN. ENOC will keep GGUS informed who in turn will keep users informed. ENOC will keep other relevant NOCs informed via TTs It is a new fault and the weathermap does not locate the fault. In this case the ENOC will query each NOC involved in the link and ask them to check the status of their portion of the link. The ENOC will issue periodic trouble tickets to GGUS until the fault is located. Once it is located , the NOC where the fault is located will send periodic TTs to ENOC who will dispatch them to GGUS and other relevant NOCs. ENOC gives feedback to close TT in GGUS, if fault is resolved

WF-3: : NREN-A notifies the ENOC of a fault • NREN-A will notify its user base following its own national procedures. • NREN-A will notify the ENOC via a trouble ticket. • The weathermap will be updated showing a fault in NREN-A • The ENOC will notify all other parties affected by the fault via a trouble ticket to the GGUS • NREN-A will send periodic updates on the resolution of the fault to the ENOC and these will be passed on by the ENOC to all other parties affected until the fault is resolved. At that time the weathermap is updated

WF-4: NREN-A notifies a scheduled maintenance • • NREN-A notifies the ENOC of a scheduled maintenance with an agreed notice period (for example 1 week). The ENOC re-issues the trouble ticket to all other relevant parties with an agreed notice period The ENOC stores the information in a DB that can be consulted by all parties via a web portal The weathermap will be able to show that a scheduled maintenance is due on a link within a period of 1 week

WF-5: The ENOC, via the weathermap, notices a fault • • • The ENOC contacts the NREN-A to clarify the situation If the fault is confirmed, ENOC informs GGUS via a TT and other relevant NOCs NREN-A will issue periodic TTs to ENOC who will dispatch them to GGUS and other relevant NOCs.

LHC End User is customer of reports general fault T 0/T 1 staff is customer of reports WLAN fault GGUS/TTM Other. . NOC NREN. . NOC GÉANT 2 NOC Trouble Tickets is customer of fault & maintenance notification reports WLAN fault ENOC Weathermap LHC View TT System MSR LHC-OPN Provides monitoring data automated perf. SONAR framework

Relevant work in GN 2 e 2 e lightpaths T 1 NREN GEANT 2 T 0 • The GN 2 JRAs (JRA 4 and JRA 3) are looking at aspects of monitoring the status of lightpaths

Draft E 2 E Link data model To be renamed “ID_Link”

Use perf. SONAR • Re-uses software developed to date • Will automatically have AA capabilities • Will allow for easier interworking with Internet 2 & ESnet • Maintenance/support provided

E 2 E visualisation and NREN services

Expectations from NRENs • It will be incumbent upon the NRENs to provide the status information of the NREN_links (and possibly ID_links) for which they are responsible. This implies: – Monitoring of the relevant alarms raised by NEs or an NMS or the periodic polling of parameters from individual NEs (directly or via an NMS) and, if needs be, concatenation of multiple measurements to provide (synthesize) the NREN_link and/or ID_link information – Establishing a service which listens for the perf. SONAR requests and responds with perf. SONAR answers • Does it mean that every NREN has to develop all this? – No! , see next slides

Option 1 • • The NRENs develop the data retrieval, local processing and stores the results in XML format , updated in (near) real time NOTE: – monitoring proxy • The NREN deploys a service (that will be provided by GN 2 JRA 1) which, when receiving a perf. SONAR request, parses the XML file and returns a perf. SONAR reply to the requester • The NREN can use any programming language to retrieve the data, analyse it and write the results into the register/XML file

Option 1 • NREN is in charge of retrieving the data from the NMS/DB, processing it and writing the data into the register file • JRA 1 to produce the service code and maintain it. The service reads the register file and sends valid perf. SONAR responses to requests (end April, early May). • JRA 1 ensures that the JRA 4 data model is integrated within the perf. SONAR schema • JRA 4 is in charge of the E 2 E NOC visualisation (and other) tools

Option 2 • The NRENs develop the data retrieval, local processing and passes the results to a JRA 1 java class which will store the data into a my. SQL database (JRA 1 provides the DB schema). • The NREN deploys a service (that will be provided by JRA 1) which, when receiving a perf. SONAR request, fetches the required data from the my. SQL database and returns a perf. SONAR reply to the requester – • Requests for instantaneous and historical data supported The main advantage of this solution is that it minimises the development to be done by the NREN. The NREN can use any programming language to retrieve the data, process it and pass the results to the java class (provided the latter is done using a predefined format).

Option 2 • NREN in charge of retrieving the data from the NMS/DB, processing it and passing the results to a java class • JRA 1 to produce the “my. SQL MA service” code (end of April) and maintain it, plus the DB and its schema • JRA 1 ensure that the data model is integrated within the perf. SONAR schema • JRA 4 in charge of the E 2 E NOC visualisation (and other) tools

Possible future options • This probably means into Gn 2 Y 3 (from September 06) • Direct SNMP access to NREN equipment • NREN data available in RRD or My. SQL DB • NMS northbound interface

The weathermap • Visualisation tool that uses the data made available by perf. SONAR • Work by DFN • A sample weathermap for POC from using NMS traps (GEANT 2 view only)