EPICS An Update on Automatic Regression Testing at

EPICS An Update on Automatic Regression Testing at Diamond

Contents 1. Motivation 2. Update on the Testing Framework at Diamond 3. Hardware Regression Test 4. EPICS Base Regression Test 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

Introduction EPICS • Diamond is now operating for ~4500 hours per year which will increase to ~5500 hours per year over the next two years • Shutdown time is being reduced, and so are becoming increasingly busy – 2010 and 2011 see installation “Mini Beta Optics” in two long straight sections necessitate replacing pairs of SR girders • All this results in less time to test new software releases • Drive to improve operational reliability – The Control System has not had significant number of faults – Don’t want to change this 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

The Grand Objective EPICS • Test all 200 support modules – Test modules and EPICS DB templates • Test device/drivers that interface to hardware – Hardware test environment • Test the versions of EPICS base we are running – Issues of changing EPICS base and /Tool chain/OS/ version • Test software against a tool chain, build and run environment – We want to ensure that operational system use the same environment

Good Software Engineering Practice EPICS • Four major parts to software engineering – Design: Should come first! – Code: We are good at this. – Documentation: Doxygen etc. – Testing: Often manual and hard to repeat. • Testing is not an optional extra – Can be carried out by the Software Developer – Or can be carried out by the Software User – Or can be automated • Tests must be properly repeated – At least before every release and preferably more often • Tests are easier to repeat if they are completely automatic 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

Test Framework EPICS • A software framework that aids the creation of automatic test suites for modules. • Based on Py. Unit, the Python unit testing library. • Test reports conform to the TAP protocol and/or JUnit XML format. • Monitoring of EPICS database records for coverage reporting. • Hardware and simulation targets can use same test suite. 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

The Run Script EPICS • Searches a directory tree for modules with automatic test support. • Runs tests suites it finds, logging results and output. • Can run multiple tests in parallel subject to resource constraints. • Level of diagnostic output can be specified. • Tests run can be restricted to single module, single target, single case. • “dls-run-tests -h” for command help. • The run script is designed to operate from the ‘support’ directory of the work tree. 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

Directory Structure Changes support new directories etc test make. Documentation make. Iocs 03/06/10 Mark Heron iocs An Update on Automatic Regression Testing at Diamond tests. py EPICS

Device Simulations • Based on existing DLS practice. • Written in Python. • Currently capable of simulating any device that communicates through serial or IP connection. • Support for instrumentation to allow protocol coverage reporting. EPICS

Framework Class Diagram 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

Example – The Test Suite from pkg_resources import require('dls. autotestframework ') from dls. autotestframework import * class Fw 102 Test. Suite(Test. Suite): def create. Tests(self): # Define the targets for this test suite Target("simulation", self, ioc. Directory="example", ioc. Boot. Cmd="bin/linux-x 86/stfw 102 Ex_sim. boot", epics. Db. Files="db/fw 102 Ex. db", sim. Devices=[Sim. Device("controller 1", 9016)], gui. Cmds=['edm -m "P=FGZ 73762, M=: WHEEL 1" - eolc -x data/fw 102. edl']) Target("hardware", self, ioc. Directory="example", ioc. Boot. Cmd="bin/linux-x 86/stfw 102 Ex. boot", epics. Db. Files="db/fw 102 Ex. db", gui. Cmds=['edm -m "P=FGZ 73762, M=: WHEEL 1" - eolc -x data/fw 102. edl']) # The tests Case. Local. Increment. Switch(self ) Case. Local. Decrement. Switch(self ) Case. Power. Off. On(self) Case. Pv. Increment(self) Case. Pv. Decrement(self) Case. Pv. Multiple. Increment(self ) Case. Pv. Multiple. Decrement(self ) Case. Pv. Set. Position(self) Case. Pv. Trigger. Mode(self) Case. Pv. Polling(self) 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

Example – Intermediate Class class Fw 102 Case(Test. Case): def cur. Device. Pos(self): ''' Get the current wheel position from the device simulation ''' result = 0 self. command("controller 1", " getpos") args = self. recv. Response("controller 1", "pos", 1) if args is not None: result = int(args[0]) return result def verify. Position(self, intended): ''' Verify that the wheel is in the intended position ''' if self. simulation. Device. Present("controller 1"): self. verify(self. cur. Device. Pos (), intended) self. verify. Pv("FGZ 73762: WHEEL 1: POSITION_RBV", intended) self. verify. Pv("FGZ 73762: WHEEL 1: POSITION", intended) self. verify. Pv("FGZ 73762: WHEEL 1: INPOS", 1) 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

Example – A Test Case EPICS class Case. Local. Increment. Switch(Fw 102 Case): def run. Test(self): '''The local increment switch. ''' if self. simulation. Device. Present("controller 1"): # Check the current position of the wheel before = self. initial. Position() # Take the wheel round twice for i in range(12): # Now advance the wheel using the local button self. command("controller 1", "incr") after = before + 1 if after > 6: after = 1 # Check the final position of the wheel self. sleep(2) self. diagnostic("Before=%d, after=%d" % (before, after), 1) self. verify. Position(after) before = after 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

Example - Test Report [1] 1. . 10 [fgz 73762@pc 0054 diamondtest]$. /runtests. py -t simulation -f default. config -i -g -b -p 3 -l tests. log -q -m fw 102 [1] ok 1 - The local increment switch. [1] ok 2 - The local decrement switch. [1] ok 3 - Power off and on. [1] ok 4 - The PV increment command. [1] ok 5 - The PV decrement command. [1] ok 6 - Fast increment sync correction. [1] ok 7 - Fast decrement sync correction. [1] ok 8 - Movement directly to a position. [1] ok 9 - Control of the trigger mode. [1] ok 10 - Control of the polling mechanism. [1] # =============== [1] # Passed 10/10 tests, 100. 00% okay, in 260. 57 s [1] # =============== [1] # Sim device controller 1 coverage report: [1] # setpos: ok [1] # settrigger: ok [1] # getpos: ok [1] # gettrigger: ok [1] # =============== [1] # EPICS database coverage report: [1] # FGZ 73762: WHEEL 1: TRIGGER(mbbo): ok [1] # FGZ 73762: WHEEL 1: TRIGGER_RBV(mbbi): ok [1] # FGZ 73762: WHEEL 1: RESTART(ai): not touched [1] # FGZ 73762: WHEEL 1: INPOS(calc): ok [1] # FGZ 73762: WHEEL 1: REINIT 1(fanout): ok [1] # FGZ 73762: WHEEL 1: POSITION(mbbo): ok [1] # FGZ 73762: WHEEL 1: COMMSFAIL(bi): ok [1] # FGZ 73762: WHEEL 1: POSITION_RBV(longin): ok [1] # FGZ 73762: WHEEL 1: CALCCOMMS(calcout): ok [1] # FGZ 73762: WHEEL 1: DISABLEPOLL(bo): ok [1] # FGZ 73762: WHEEL 1: STEPBK(calcout): ok 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

Hudson • Use Hudson framework – For scheduling and reporting of test • A Continuous Integration Engine – Checkout from repository – Does some actions • Build, Post-build actions, (execute test script) – Determine Status • Invoked by period checks on SVN repository • Status by web and emails to developers • Success (all actions successful) • Unstable (some tests failed) • Failed (compile error) 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

EPICS 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

Hardware Regression Test EPICS • Test IOC consists of: – Prodex OMS-58 8 axis servo control card • A voltage to quadrature encoder conversion card – Hytec 8001 DIO and 8515 Octal Serial – EPICS IOC test specific db • Stimulation IOC: – 8005 – digital I/0 for limit switches, home switches – 8401 – ADC input card to monitor OMS-58 servo demand output – 8402 – DAC to drive the encoder simulation – Vx. Works application implemented by cut down drivers for above cards and a simulation of some ‘realish’ hardware to control • Test harness: – Wiring to join Test IOC to Stimulation IOC. 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

Hardware Test Framework EPICS Test harness Switch Serial Stimulation Unit Test Unit (EPICS IOC) Hardware under test Terminal Server Digital & Analog I/O Processor (vx. Works stimulations) Processor Power strip (IP) Linux control PC Test suite control, Hudson 03/06/10 Mark Heron redirection, etc An Update on Automatic Regression Testing at Diamond Network

EPICS Base Regression Test Suite (1) • Came out of the Codeathon at Diamond – Nick Rees, Jon Thompson and Matt Pearson • Objective – To generate automated testing of the EPICS Base project code on a diverse range of architectures, both workstation and real-time OSs. – These will be run nightly at Diamond on head of trunk and results published on web site. – Provide as service to the collaboration • This is work in progress. Only started it last week. 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

EPICS Base Regression Test Suite (2) • There are established EPICS tests, but these are run manually as part of releases • Wrap the established EPICS tests in Python to make callable from test framework. • Run test on – EPICS Base, C Unit tests – Report on test coverage for each EPICS Base test – IOCs based tests 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

EPICS Base Types of tests (1) EPICS • C unit tests based on epic. Unit. Test. h functions. – These are mostly used to test lib. Com functions at this point – Generate TAP output that is interpreted by the test harness – Propose to add new make targets to describe how the tests are run on embedded architectures (give a download binary and a command line function to run) – Make generates a test specification that is used by the test framework to run the tests 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

EPICS Base Types of tests (2) EPICS • IOC based system tests based on Diamond module test framework – Python framework that loads IOC under test and a stimulation system (if required) – Can get, set and monitor PV’s, and/or control or monitor the IOC command line – Tests reside in a new etc/test directory in the application tree – Have a wide range of verification commands. – Many of the EPICS “soft-test” suite tests were converted to this framework during the Codeathon 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

EPICS Base Platforms EPICS • Plan is to run this for popular development platforms and target architectures. • What about historic development platforms, Solaris, and latest platforms say Fedora XX or Ubuntu XX. YY and other target architectures ? 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

EPICS Base Test Results EPICS • Results will be published as a web page containing: – Results of ‘make runtests’ on the target architectures – Results of running the ‘soft-tests’ on the target architectures – Code coverage report for the Linux host. • URL to be announced 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond

EPICS Base Test Results Web Interface 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

EPICS Base Test Results Web Interface 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

EPICS Base Test Results Web Interface 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond EPICS

Acknowledgement EPICS • This is being lead by Jon Thomson at Diamond • It’s the work of many of the Controls Group at Diamond 03/06/10 Mark Heron An Update on Automatic Regression Testing at Diamond