What does QA mean to Sakai Megan May

What does QA mean to Sakai? Megan May – Sakai Foundation Aaron Zeckoski – CARET Alan Berg – UVA David Horwitz – Cape Town Seth Theriault- Columbia Linda Place – U of M

Today’s Session • • What is QA? Unit/integration testing Static code review Release/Branch management QA server network QA WG Performance testing 2

What is quality assurance? the set of planned and systematic actions necessary to provide appropriate confidence that a product or service will satisfy the requirements for quality. What is the role of the QA Director? 3

Unit/Integration Tests Aaron Zeckoski

Test Driven Development • Create the Class/API -> Write the Test -> Program the Implementation -> Run the Test • Forces you to use your own method (and hopefully check if it is intuitive to use) • Requires an immediate check against the javadoc (API) • Makes the developer think about how the method will work and what it does BEFORE they write any code • http: //en. wikipedia. org/wiki/Test-driven_development 5

Bug fix test cases • When there is an issue created against your code you should create a test case to test for that specific failure before you fix it • These steps – Create a test case which fails as indicated in the issue from the issue tracker (JIRA) – Reference the JIRA URL in the comment for the test case – Run the test and make sure it fails – Fix the problem – Rerun the test to make sure it passes – Now you are safer from regressions 6

Tests vs. Code Review • Static code review is valuable and helpful • Peer code review is more valuable • Neither is a substitute for unit testing and integration testing though • Why? – They do not provide for protection from regression – They do not supply the context that tests do • Tests show to use the code and provide a type of documentation 7

Static Code Review Alan Berg

Static Code Review Functional testing – – – Guarantees a minimum quality Black box looks from the outside in Does not cover all the code Labor intensive Does not totally scale as the code base scales 9

Static Code Review • Static Code Reviews – – – – Looks at the source code not a running instance White box, looks from the inside out Looks through all the code base once a day Automated Steps over commit boundaries Information overload Requires the right mentality from the community Need to eat the bugs with little fuss 10

TOUR qa 1 -nl. sakaiproject. org/codereview/bug_dashboard/index_generic. html 11 11

TOUR qa 1 -nl. sakaiproject. org/codereview/bug_dashboard/index_generic. html 12 12

Source code 13 13

Evolutionary • Using PMD, Findbugs, QJ Pro, Jlint and custom Perl code • Tools getting better • Findbugs does not have many False positives • PMD has configurable rules • More shaping the information to avoid information overload • Engaging in debate on how to deal with bugs • Need to change developers perceptions 14 14

Other • • • Code coverage Automatic removal of simple bug types script Tracking of long term use of memory Provisioning for performance testing More to come 15 15

Branch/Release Management

Branch Management & Enlightened Self interest • Why would we want to be branch managers? – first to go to 2. 5. x – wanted a say in ensuring a high quality branch for production – Its not actually a big overhead outside of the release cycle 17

QA Server Network Seth Theriault David Horwitz

QA Server Network Amsterdam Cambridge Columbia r. Smart Boston U Cape Town Indiana Georgia Tech Charles Sturt U Coming soon - server in Japan 19

Running a qa server • What you need: – – a computer a network connection a database know how to compile and install Sakai • Initial setup – Build server 20

Running a qa server • Updating and maintenance – – downloading source build deploy upgrade data (maybe) usually once a week during release 21

QA WG

WG Tools Collab https: //collab. sakaiproject. org/portal JIRA http: //jira. sakaiproject. org/ Confluence http: //confluence. sakaiproject. org/confluence/displ ay/QA/Home 23

Sakai Performance Testing Current model and limitations Desired model and benefits Dr. Linda M Place, University of Michigan

Talking Points • Current model – Black box testing – Performance failure debugging and fixing • Desired model – White box testing with baseline load – Minimize production performance failures 25

Black Box Testing • Black box testing – Running system with projected use-case scenarios for acceptable throughput and response times • How will users experience application? – Baseline comparison • Throughput and response times compared against previous version test results • Looking for as good as or better results from new version – Typically occurs at end of QA process 26

Qualities Being Tested • Push application load against a system to identify quality of performance – – Transaction response time for end users Throughput (hits per second) Reliability (percentage of failed transactions) Scalability (what happens when scale increases significantly) – Capacity (upper limits of system) 27

Results • Are metrics being met? – Measures of success • Current Sakai test metrics – Baseline or better response times and throughput – New transactions do not disrupt baseline metrics – New transaction stress tests • How many occurrences before system becomes unusable? 28

Failed Performance Debugging • Designing isolated tests to address obvious performance problems in production code – Use tests to isolate performance cause – Use tests to verify fix before adding to production instance • Putting out fires 29

White Box Testing • Pushing system to identify application, database, operating system, or network problems – Tune environment to identify and address specific problems – Tests watched by developers, DBA, system and network administrators, and performance engineers • Typically occurs during code development 30

Little in Sakai Today • Developers may design performance tests as part of development – Process not codified as required standard for development – Most developers do not have a system of sufficient scale to push code adequately – All components of production environment not used and monitored • Developers; database, system & network administrators; performance engineer 31

Value to Sakai • Adding white box testing into Sakai QA process finds and fixes numerous performance problems before they reach formal QA process • Engages Operations team early in development cycle rather than after performance problem affects user community • Debugging easier prior to full integration with application suite 32

Want White Box Testing? • Stay in room for next session – Sakai Community Performance Testing: A Working Proof of Concept • Participate in community model to improve your code prior to subjecting it to QA • Don’t expect someone else to identify and fix your codes performance shortcomings 33

Questions 34