Towards Further Automation of the Quality Assurance Cycle

Towards Further Automation of the Quality Assurance Cycle Alan Berg: University of Amsterdam Central Computer Services (IC) Group Education and Research Services

But first an advert ’ ing QA sexy you akes m

QA or the Sakaiger gets it

QA’ing helps you get the toy

We have only 30 minutes, so only a brief biased personal snapshot is possible

Agenda • The potential for a QA event horizon • What does the QA WG want? • Current hints to the developer – Where are we • The Sash Tool • Jameleon • Conclusions

The QA event horizon Risk losing a deterministic QA cycle http: //amazing-space. stsci. edu/resources/explorations/blackholes/teacher/grabbag. html

To remain deterministic we will need more automation

The QA event horizon Risk losing a deterministic QA cycle The Obvious • Code base increasing rapidly • Not just Java, but also XML, Javascript, Python, CSS and many Java frameworks • Iterative process that requires defined response times • Requires manpower/resources on time • Would love QA as part of Continuous Integration

What does QA WG want? • A "stack" of open source tools for load testing and nightly integration testing • A load-testing repository and "cookbook" for open source tools • Regression testing work flow • QA Process - no tool makes it into the core (stable) set of tests • QA Process - no bug in JIRA is closed without an automated test that checks for that bug. http: //bugs. sakaiproject. org/confluence/display/QA/Q A+Test+Automation

Hints to the developer

Supporting developers • Nightly – builds – Continuum [Claret] – Javadocs - [nightly] – Static code reviews – [Uv. A] – Internationalisation - [LOI ] – Unit testing [ developers ] • – Potentially through Continuum • -Mock objects from Josha Holtzman • Knowledge capture [Confluence, community, conferences]

Hinting nightly http: //qa 1 -nl. sakaiproject. org

Random Example

Duplicate Code

Do we wish to use the reports more aggressively? • Are the developers taking note? – Bugs reported are not always correct – Bugs found not always important – But duplicate code, unit test coverage, and failing to deal properly with exceptions hint strongly at quality. • Answer probably not, but thankfully trends in the marketplace will improve the quality of results overtime.

The Sash Tool

The Sash Tool A W S O M E http: //qa 1 -nl. sakaiproject. org: 8080/

Realistic Proof of Concept • Provision tool – add/delete users, sites, members, tools, permissions – upload files from file system to dropbox/resources – Controlled by a property file in site resources • • live. Test Diagnose Report But have we enough API’s to do the job?

SASH TOOL (Steven Giffens) Live Brilliant for learning Environment that targets sys_admin Easy to extend commands Consistent use of cover api's allow services to be easily guessed. • Use to write provision, diagnose, live test tools. • • • Just requires imagination and Sakai specific knowledge.

import org. sakaiproject. tool. cover. Tool. Manager; import org. sakaiproject. tool. api. Tool; import org. sakaiproject. component. cover. Component. Manager; import org. sakaiproject. content. api. Content. Hosting. Service; LIVE ident=". . "; all = Tool. Manager. find. Tools(null, null); counter=1; for ( Iterator i = all. iterator(); i. has. Next(); ) { tool= i. next(); print("["+counter+"] "+tool. get. Title()); print("ID : "+tool. get. Id()); print(ident+tool. get. Description()); print (ident+ident); counter++; } counter=1; print ("Interfaces"); inters = Component. Manager. get. Registered. Interfaces(); for ( Iterator i = inters. iterator(); i. has. Next(); ) { inter= i. next(); print("[: "+counter+"]"+inter); counter++; } Trunk 2. 4. . Tools=104, Interfaces=568 live Uses covers Consistent use of java. utils

Cover like. . . import org. sakaiproject. test. cover. Test. Manager; import org. sakaiproject. testl. api. Test; all = Test. Manager. find. Tests(Test. Manager. RUNALL); counter=1; for ( Iterator i = all. iterator(); i. has. Next(); ) { test= i. next(); print("["+counter+"] "+test. get. Title()); print(test. get. Description()); try{ print(test. do. Test()); }catch(Failed. Test. Exception e){ print(“Error was: “+e. get. Message()); }finally{ counter++; } }

Remember QA Process - no tool makes it into the core (stable) set of tests BRAINSTORM ONLY • Push tests back to the developer • Runs against a live system • Can be removed at compile time for production distributions • Non standard [Yeuch] • Consistent, and test driven • Easy to script through Sash • Only solves a certain significant class of tests

Jameleon http: //jameleon. sourceforge. net/

Regression testing workflow CRITERIA • Run automatically • Results consistent • Many roads to Rome. – Selenium RC wrapper in Maven, Python, Ant • Cost benefit ratio favourable

Jameleon - Proof of Concept • Wrapper around Selenium RC and other tools – Junit, Jiffie (Internet Explorer, Http. Unit, generic) • • • Configured via XML Generates reports Can run headless via cron or continuum Data driven Almost self documenting

Data driven • Makes tests server generic • Property file per organization – uva. properties • name=admin • pass=admin • base=qa 1 -nl. sakaiproject. org: 8380 • csv with variables defined in first line • eid • newbiex • newbie 2

Self documenting http: //jameleon. sourceforge. net/

XML test cases Add user to Sakai. QA Alan Berg SMOKE Add User to Sakai ${base. Url}

Workflow • From Cron – Download and build Sakai – Copy different scripts to directory – Run Jameleon • (can run against IE, Firefox, Opera etc) – Generate reports – Move reports to website • Can be added as a script to run to continuum

Results Works Selenium plug-in still beta Tests are fragile and require maintenance Will require a significant effort for basic coverage • Still very promising and should be considered. • HATS OFF TO THE DEVELOPERS • •

My plans • I have one day a week free to: – Concentrate on Sash scripts – Wait until the next release of Jameleon. – Beg the big brains to build a realistic test manager.

? ? ? Questions