IRMIS Crawlers Tutorial Presentation by Debby Quock Janet

IRMIS Crawlers Tutorial Presentation by Debby Quock (Janet Anderson) IRMIS Workshop EPICS Collaboration Meeting 2008 INFN - Laboratori Nazionali Di Legnaro

IRMIS 2 User Interface User Viewers § Global Search Tool § AOI § IOC § PLC § Component Type § Network Info § Racks Info § Controls Spares § Component History User Editors § AOI § PLC § Component History § Installed Components § Cables

Other ways of getting data into the IRMIS database… …Automated Crawlers 7 Main Crawlers to Date § ADL • EPICS user displays § ALH • EPICS alarm handler § AOI • Applications Organizing Index § Network § PV § SDDS • Self-Describing Data Sets § Sequence • EPICS sequence programs

IRMIS 2 Database Schema aoi Ø 19 tables automatically populated by crawlers fld aoi_crawler 76 IRMIS Tables component_semaphore component_state fld_type aoi_criticality component_state_category machine aoi_document component_type mfg aoi_epics_record component_type_document person aoi_ioc_stcmd_line component_type_function person_group aoi_machine component_type_if plc aoi_note component_type_if_type plc_old aoi_plc_stcmd_line component_type_person port_pin aoi_status component_type_status port_pin_designator aoi_techsys conductor port_pin_template aoi_topdisplay criticality_type port_pin_type aps_component doc_type rec aps_ioc fld rec_client audit_action fld_type rec_client_type audit_action_type form_factor rec_type base_component_type function rec_type_dev_sup cable group_name role component ioc role_name component_instance ioc_boot server component_instance_state ioc_error technical_system component_port ioc_error_message uri component_port_template ioc_resource vuri component_port_type ioc_resource_type vuri_rel component_rel ioc_stcmd_line component_rel_type doc_type

IRMIS Crawlers Software Framework § Blowfish database connection encryption § Logs § Cronjob execution SHELL=/bin/sh MAILTO=quock@aps. anl. gov 0, 30 * * /local/home/IRMIS/pvcrawler/run_crawler. sh 20 1 * * 6 /local/home/IRMIS/alhcrawler/run_crawler. sh 25 2 * * 6 /local/home/IRMIS/adlcrawler/run_crawler. sh 50 1 * * 6 /local/home/IRMIS/sddscrawler/run_crawler. sh 15, 45 * * /local/home/IRMIS/sdds. PVdump/run_crawler. sh 10 8 * * * /local/home/IRMIS/networkcrawler/run_crawler. sh 0 4 * * 6 /local/home/IRMIS/rec_client_mainenance/run_clean. sh § Command line execution Example: PV Crawler pv_crawler –go –boot-scan=<module-name> [--no-sequence-scan] [--key=<16 char key>] [--test] [--force] [--help] [--debug-level=…] § Perl modules Example: ADL Crawler § § § ADLCrawler. DBLayer. pm ADLCrawler. Parser. pm adl_crawler. pl APSAdl. Directory. File APSEdp. Directory. File Blowfish_PP. pm db. properties Log. Util. pm My. SQLDBFunctions. pm Oracle. DBFunctions. pm run_crawler. sh

IRMIS Crawler Requirements § Perl version 5 • 5. 8. 8 in use at APS § Perl modules • DBI (Database Interface module) • DBD: : mysql (My. SQL driver for DBI) § My. SQL version 5 • 5. 0. 27 in use at APS § EPICS Macro Substitution and Include (MSI) tool § Reasonably consistent IOC boot process § Some ability to write/edit Perl code • To create *Boot. Scan module • To handle site-specific boot peculiarities

IRMIS PV Crawler Mechanics db. properties pv_crawler. pl path. properties D B I Parser *Boot. Scan ioc_boot, ioc_resource rec, fld, … My. SQL bootparams st. cmd db IOC boot file system(s)

IRMIS PV Crawler Mechanics, Cont. Once every 30 minutes, the PV crawler: – Identifies IOCs to be scanned and their last known boot date (i. e. , APSBoot. Scan. pm or SNSBoot. Scan. pm) – For each IOC • Checks for new boot date • If so, insert new ioc_boot record in db and proceed • If not, skip to next IOC • Parses st. cmd (collecting all info on dbd and db files) • Checks for any dbd or db file time mods • If so, insert all pv and field data in db for IOC

IRMIS PV Crawler Core DB Schema PV Crawler modifies tables: ioc_boot ioc_error ioc_resource rec_type_device_support fld_type uri

IRMIS PV Crawler Stored Information – Boot history • Each new IOC boot is recorded with date, regardless of whether any configuration has changed • If there is an identifiable problem with the boot, or inability to parse its configuration, an associated error is recorded – File system resources (only if changed from last crawl) • Path to each dbd and db file that make up configuration, along with file modified date and any text substitutions applied – Record type definitions (dbd) (only if changed from last crawl) • The set of fields defined for each record type at that time – Record definitions (db) (only if changed from last crawl) • Record name, fields and their default/overriden values

Example PV Crawler Log File Output I|20080723153104|Processing iocbdg 2, boot date: 20080115074215 I|20080723153104|Parsing /usr/local/iocapps/R 3. 10/ioc/booster/3/ioc. Boot/iocbdg 2/st. cmd I|20080723153111|IOC Resources changed: 0, Parse error: 0 I|20080723153111|Write data to DB I|20080723153111|No sequences found for this IOC I|20080723153111|SEQCrawler. DBLayer: update_db_tables finished. I|20080723153111|Processing iocbol, boot date: 20080425103028 I|20080723153111|Parsing /usr/local/iocapps/R 3. 10/ioc/booster/3/ioc. Boot/iocbol/st. cmd I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 1, name=m. Init. MEM 1" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 2, name=m. Init. MEM 2" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 3, name=m. Init. MEM 3" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 4, name=m. Init. MEM 4" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 5, name=m. Init. HEM 5" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 6, name=m. Init. HEM 6" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 7, name=m. Init. HEM 7" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153122|SEQ_PV seq command is seq &motor. Init, "M=BOO: Opt. Lab: M 8, name=m. Init. HEM 8" I|20080723153122|SEQ_PV Sequence source file is /usr/local/iocapps/R 3. 10/ioc/booster/3/bol. App/src/motor. Init. st I|20080723153122|SEQ_PV Number of sequencer pvs is 17 I|20080723153123|IOC Resources changed: 0, Parse error: 0 I|20080723153123|Write data to DB I|20080723153124|SEQCrawler. DBLayer: update_db_tables finished. I|20080723153125|Processing iocbramp, boot date: 20080527111355

AOI Crawler Discovered Attributes n AOI Crawler parses IOC EPICS startup command files to retrieve information about: - UPCs (IOCs, PLCs, Lab. View, . . . ) associated with each AOI - Process Variables associated with each AOI - EPICS startup command lines for later viewing directly through IRMIS sequence programs configuration data n ld < mps. Tester. Single. Latch. Card. o Mpc. Config 0, 0 x 3100, 80, 3 Parsing requires unique AOI comment lines in the EPICS IOC startup command files: # load one or more databases here #<aoi name="aoi_linac_llrf_control_l 6"> db. Load. Records "linac. App/llrf. Db/llrf 6. db" #</aoi> #<aoi name="aoi_linac_rf_modulator_l 6"> db. Load. Records "linac. App/modulator. Db/modulator 6. db" #</aoi>

AOI Crawler DB Schema AOI Crawler updates tables: aoi_ioc_stcmd_line aoi_plc_stcmd_line ioc_stcmd_line aoi_epics_record aoi_crawler

Sequence Crawler Discovered Attributes n In IOC startup command files, searches for script lines: – – ld (load) register. Record. Device. Driver #! (shebang) seq (sequence) n Utilizes information: – – IOC architecture executable sequence file sequence library file EPICS database load file n Performs tasks: – – Marks sequence records as old by setting table field rec_client: : current_load =0 for any records associated with the current IOC Inserts new rows into tables uri, vuri_rel, and rec_client. Table rec_client holds PV name and points to table vuri. Table uri holds the file name for the sequence program.

Sequence Crawler DB Schema

ADL Crawler Discovered Attributes n Parses MEDM adl files and looks for PV names n Modifies tables rec_client, uri, vuri_rel with adl file names and PV names n Execution command adl_crawler. pl --go --adl-dirs=APSAdl. Directory. File --edp-dirs=APSEdp. Directory. File --key=***** >> logs/crawler. log 2>&1

ALH Crawler Discovered Attributes n Parses alarm handler alh config files for IOC record references n Writes file name and PV name to tables rec_client, uri, vuri n Execution Command alh_crawler --go –alh-dirs=<file-name> [--key=<16 char key>] [--test] [--help] [--debug-level=[verbose | debug] info]]

Network Crawler Discovered Attributes n Reads a file of network switch interface data and determines if there are changes to the IOC primary and secondary Ethernet switch, blade, and port settings n Writes IOC network interface data located in file accel_sw to IRMIS database table aps_ioc n Network data file accel_sw is updated nightly by IT process n Network Crawler is run once every morning

SDDS Crawler Discovered Attributes n Parses SDDS files and writes any Control. Name column to the IRMIS rec_client table n Modifies tables rec_client, uri, vuri n Execution Command sdds_crawler -- go –sdds-files=<file-name> --sdds-app-name=<app-name> [--key=<16 char key>] [--test] …

IRMIS Crawlers Future Enhancements § Merge the AOI and PV crawlers into one • Code reuse and code maintenance § Separate historical EPICS PVs from current PVs • • Leaning towards separate tables for holding historical data Crawler will push PV data into historical tables when an IOC boot change has been detected § New crawlers • • IOC Save/Restore Controlled Access System (CAS) § Enhancements to existing crawlers • Email notification Notification to IOC cognizants in the event of Crawler discovery of IOCrelated errors. Same for AOIs. • ADL Crawler Revise ADL crawler to search for obsolete PVs and search adl files for "adl cognizant". Notify cognizant via email that adl file needs updating to remove or change obsolete PVs.

Acknowledgements § § Don Dohan, Brookhaven National Laboratory Ned Arnold and Claude Saunders, Argonne National Laboratory Mariana Varotto, former ANL employee Ron Mac. Kenzie and Judy Rock, Stanford Linear Accelerator Center