Software Diagnostics and Conformance Testing CRT Voting system

Software Diagnostics and Conformance Testing CRT: Voting system logic testing (Votetest) David Flater 2008 -01 -23

Players and field Software Diagnostics and Conformance Testing • Congress – – • Help America Vote Act (HAVA) National Voting Rights Act Section 508 Americans with Disabilities Act Election Assistance Commission (EAC) – Voluntary Voting System Guidelines (VVSG) – Manufacturer registration, lab accreditation, certification, … • • Technical Guidelines Development Committee (TGDC) NIST – National Voluntary Lab Accreditation Program (NVLAP) • • Voting system manufacturers Voting jurisdictions State and local election officials Concerned citizens Professional advocates Academics Reporters & bloggers (and they all have lawyers)

Software Diagnostics and Conformance Testing Logic testing in context • • Manufacturer-driven activities Conformity assessment – Physical configuration audit – Documentation and design reviews – Electromagnetic compatibility and environmental testing – Logic testing (Votetest) – Volume test (mock election) – CRT benchmarks – STS and HFP testing • • Election Assistance Commission (EAC) certification Jurisdiction acceptance testing and certification Deployment Monitoring

Software Diagnostics and Conformance Testing Goals • • • Status quo: test labs are on their own to develop conformance tests for the Voluntary Voting System Guidelines (VVSG) Conservative goal: reduce variability and cost of testing by providing test labs with tools and materials useful in constructing test suites Ambitious goal: further reduce variability and cost by providing a canonical test suite

Software Diagnostics and Conformance Testing Choosing the right tools • “Testing target: ” the object of conformity assessment – A. k. a. Implementation/Device/System Under Test • Different kinds of testing targets need different testing approaches

Differences from other testing targets Software Diagnostics and Conformance Testing • Automatic testing is not feasible – Don’t have standard interfaces to get data in and results out – Voters are part of the process (people in the loop) – Unanticipated nonfatal errors must be detected • Cost of executing tests is a major issue – Significant time and effort to prepare election definitions, ballot styles, and test ballots or voters for each test case – Labor costs for people in the loop – Politics: any increase in total cost for certification will be considered an unfunded mandate • More is not better – A vote is a vote (logically) – As we increase the number of votes counted • Cost of testing increases proportionally • Return on investment diminishes rapidly • Context: one step in a long process – Volume test (mock election), logic verification, etc.

The requirements Software Diagnostics and Conformance Testing • • • Normative reference: the next iteration of the VVSG (in public review) Logic must correctly handle all voting variations that the manufacturer claims to support Everything must work through the complete elections and voting process Election definition Ballot definition Configuration and calibration of equipment Logic and accuracy testing Vote gathering Tabulation Reconciliation Reporting 1 of M voting X X X X N of M voting X X X X Cumulative voting X X X X Ranked order voting X X X X In-person voting X X X X Absentee voting X X X X Provisional / challenged ballots X X X X Write-ins X X X X Review-required ballots X X X X Primary elections X X X X Split precincts X X X X Ballot rotation X X X X Straight party voting X X X X Cross-party endorsement X X X X

Testing strategy Software Diagnostics and Conformance Testing • • All tests are end-to-end tests that exercise the complete elections and voting process Small number (10 -100) of carefully selected tests – Cover each voting variation with a simple, synthetic test (around 10 ballots, 1 contest) – Similarly cover all meaningful pairs of voting variations – Few slightly larger tests (around 100 ballots, multiple contests) based on real sample ballots – Few miscellaneous tests (e. g. , boundary cases) • • • Test scripts to be “realized” according to the specifics of the target Test oracle No big tests in this test suite – Context: The big volume test (mock election) provides a significant test of all supported voting variations together • Punt devilish details – – Some requirements are too implementation-dependent Some requirements are incidental to every scenario Provided test descriptions but not test cases Test lab is responsible for complete coverage

Votetest release strategy Software Diagnostics and Conformance Testing • First release – – • Based on draft VVSG “Basic test suite” Tools and materials Needs review and feedback Second release? – If consensus is that basic test suite is not enough – If there are problems to correct – Sync with finalized VVSG (if applicable) • Maintenance and support – Keep up with VVSG maintenance (interpretations, errata) – Correct operational issues and coverage gaps as they arise

Software Diagnostics and Conformance Testing Votetest contents • Data model that supports all draft VVSG voting variations • SQL* schema that realizes the data model and the tabulation logic specified in the draft VVSG • Test cases formalized as SQL scripts – – • • We don’t know the interface to the test target SQL used as surrogate language Execute as written on the supplied database Must be translated into whatever is required by the test target Report generator to display results from test oracle Expected test results Documentation Bonus: test generator * Schema uses extensions to ISO SQL

Test case execution Software Diagnostics and Conformance Testing Votetest environment Test case (SQL) Database Translate Report generator Expected results Compare Test case (translated) Voting system Report generator Voting system environment Actual results

Usability of logic test tools and materials Software Diagnostics and Conformance Testing • • Technical expertise befitting an accredited test lab is assumed and required Test cases formalized as SQL scripts – More precise than informal test scripts – Automated translation is possible • The expected output from each test case is provided as a plain text report – Test lab does not need to get the infrastructure to run on their machines to use the test scripts – Sanity check for running installations • No huge up-front investment – Hardware requirements: one surplus PC – Software requirements: all free software

################################### BEGIN TEST CASE OUTPUT 2007 -12 -27 15: 52 -05 ################################### • • Software Diagnostics and Conformance Testing $Id: 1 -basic-1 of. M. sql 415 2007 -12 -27 16: 34: 15 Z dflater $ Small 1 -of-M contest, no write-ins, no rejected ballots. Ballot styles: 1 Reporting contexts: 1 [. . . Integrity checks deleted. . . ] [. . . View materialization log deleted. . . ] • • Print header Reset database to baseline state Load test data Run integrity checks ---------------------------------------Report for context Precinct 1 generated 2007 -12 -27 15: 52 -0500 BALLOT COUNTS Configuration ------Total Blank Precinct 1 Style Blank Read ---12 1 Counted ------12 1 • Generate report VOTE TOTALS President, vote for at most 1 Car Tay Fower 4 Tayra Tree 3 Beeso Tu 2 Oona Won 1 Nada Zayro 0 Overvotes 1 Undervotes 1 Counted ballots 12 Balance 0 ---------------------------------------Report total volume: 76 - Includes optional reporting of blank ballots. - Excludes separate reporting of ballots cast vs. read. ################################### END TEST CASE OUTPUT 2007 -12 -27 15: 52 -05 ################################### • Print footer

The oracle Software Diagnostics and Conformance Testing • • Design requirement is correctness not performance Logic model of draft VVSG translated as transparently as possible into SQL views – Limited expressiveness of SQL means fewer ways to introduce faults (vs. programming) – Good news: the logic model itself translates with minimal overhead – Bad news: straight party voting and write-in reconciliation add a level of complexity • • Informal verification of correctness included in documentation Demonstrated scalability up to 2 million ballots Results of simple tests are manually confirmed Test suite + saved output + shell script = automated regression test X

Software Diagnostics and Conformance Testing Status as of 2008 -01 -23 • • • 3 baseline tests (no optional voting variations required) 19 single-variation tests covering 12 optional voting variations 66 two-variation tests covering 63 combinations of two voting variations – The other 3 combinations are not meaningful • • • 1 three-variation test 3 tests based on sample ballots Total of 92 tests • • Working on documentation and presentation Could improve test generator and do more samples tests Needs NIST internal review, integration with other test efforts No public release yet

Software Diagnostics and Conformance Testing Challenges • • • Can’t review prior art—everything claimed as trade secret Draft VVSG is a moving target—Standards and Advisory Boards Accretive release strategy—pressure to get it right the first time Realism—no two jurisdictions are alike Politics

Demo—Disclaimers Software Diagnostics and Conformance Testing • • • For demonstration purposes only, we are about to execute a test case in an emulated environment This configuration has problems and is not recommended for production use The nonfatal error shown below should be ignored could not remove file or directory "base/55958": Directory not empty

Test case execution Software Diagnostics and Conformance Testing Votetest environment Test case (SQL) Database Translate Report generator Expected results Compare Test case (translated) Voting system Report generator Voting system environment Actual results

THE DEM 19

Software Diagnostics and Conformance Testing End of presentation