What does Quality Assurance Mean to Sakai Alan

What does Quality Assurance Mean to Sakai? Alan Berg Megan May Seth Theriault 8 th Sakai Conference Newport Beach 4 -7 December 2007

Topics • What does a QA Director do? • QA Today – Who is in the WG – Types of testing currently done and release activities – Strides forward • Resources available – QA Network • 2. 5 Release Activities Update • QA Tomorrow • Questions

Role of QA Director • Coordination! – Help bring parties interested in the same testing efforts together – Foster initiatives that support quality throughout the community – Assist in bridging the gap between test and design & development teams – Ensure release activities are transparent

QA Today Focus on Functional Testing Requirement Gathering and Design Development Testing & Software Release

Who is the QA WG? 2. 2. 1 2. 3. 0 23 People 44 People 10 Institutions 17 Institutions 4 Countries 5 Countries . . . How do they get involved? 2. 4. 0 2. 5. 0 93 People 56 People 37 Institutions 11 Countries 4 Countries

Strides Forward • Deployment testing on wider variety of platforms • Integration of Accessibility WG activities in general QA cycle • Expanding/updating library of test scripts • Reflection on Release process – http: //confluence. sakaiproject. org/confluence/x/fb. U

QA Network Amsterdam Cambridge Columbia r. Smart Boston U Cape Town Indiana Georgia Tech Coming soon servers in Japan and Australia

Collab How to be in the know! https: //collab. sakaiproject. org/portal

Jira http: //jira. sakaiproject. org/

Confluence http: //confluence. sakaiproject. org/confluence/display/QA/Home

Perceived Quality Level

Quality, like beauty, is clearly in the eyes of the beholder.

Perceived Quality Poll

QA Tomorrow • Sakai Community Concerns – Several prominent campuses experience serious performance problems with 2. 4 • 2. 5 Release Update – Further refinement of process – Goals for 2. 5

Paths Forward • Common set of testing vocabulary – – Unit Testing Integration testing System Testing Performance • Load testing (lmp@umich. edu) • Integration testing Requirement Gathering and Design Development Testing & Software Release http: //confluence. sakaiproject. org/confluence/x/M 4 B 5

Roadmap for Tomorrow

Metrics to Describe Quality 2. 2. 0 – 85% Verification (504 JIRA Tickets) 2. 3. 0 – 85% Verification (611 JIRA Tickets) 2. 4. 0 – 82% Verification (1, 345 JIRA Tickets) 2. 5. 0 – Past bug verification

Metrics to Describe Quality

Static code Review • Search for bug patterns • What to process: Source code, Binary • Two types of bug pattern – Generic • Classic (buffer overflow, input validation) – Contextual • Language specific • Framework specific • Configuration specific • Proximity effect

Hinting nightly http: //qa 1 -nl. sakaiproject. org

Random Example

Duplicate Code

Do we wish to use the reports more aggressively? • Are the developers taking note? – Bugs reported are not always correct – Bugs found not always important – But duplicate code, unit test coverage, and failing to deal properly with exceptions hint strongly at quality. – Not enough Contextual searching for Sakai and spring configuration specific issues – Reports do not explain enough – Answer probably not, but thankfully trends in the marketplace will improve the quality of results overtime.

A year on • Update prototype – Less noise more explanation – Graphical tracking of change • • Recommend Eclipse Plugins Look at emerging products See as part of a series of QA processes. Agree on metrics to act on and then act.

A good book to read • Secure Programming with Static Analysis – Brian Chess, Jacob West • Addison Wesley • ISBN: 9780321424778 • Includes CD with software and examples.

What you can do • Pledge Resources – What happens in the community has a direct local impact – Easy point of entry into Sakai – All skill sets are needed • Discussion – Why hasn’t this been a priority? – What are obstacles in the way and how do we remove them?

Questions

Other Sessions of Interest • 2. 5 QA Workshop (Thursday, 13: 00 - 14: 30 in Salon 5) • The Automated Testing Conundrum in Sakai (Thursday, 15: 10 - 16: 10 in Salon 5)