Update on SAMATE Automated Source Code Conformance Verification

Update on SAMATE Automated Source Code Conformance Verification Against VVSG 1. 0 Requirements Michael Kass National Institute of Standards and Technology http: //vote. nist. gov TGDC Meeting, December 2011

Outline n n n Progress report on automated source code conformance verification effort introduced at last TGDC meeting Lessons learned Next steps TGDC Meeting, December 2011 Page 2

Software Assurance Metrics and Tool Evaluation (SAMATE) n NIST/DHS co-sponsored project n n Goal - measure the effectiveness of software assurance tools to identify weaknesses and vulnerabilities in software Asked by EAC to assist voting system manufacturers and test labs improve a (heavily manual) source code review process by developing a uniform automated conformance verification capability for voting system source code against VVSG 1. 0 software requirements TGDC Meeting, December 2011 Page 3

Tool Automation Progress n Finished and tested automation of VVSG 1. 0 rules for Java (still documenting C/C++ work) n n 50 custom rules written for the “Checkstyle” open source Java code analysis tool 41 verification tests (sample “bad” code) against those rules, to verify tool correctness against requirements Ran customized tool against 1 M+ lines of Java source code Bundled Java tool rules and tests into ZIP and tar archive files for distribution to TGDC, VSTLs and manufacturers TGDC Meeting, December 2011 Page 4

Requirement Automation Overview n Identified 58 software requirements in VVSG (since last TGDC report) n n n 26 requirements fully/partially automated for Java 12 requirements do not apply to Java language 14 requirements require entirely human analysis 3 requirements could be automated, but require “custom” knowledge about code or documentation 2 requirements could be verified with a “more capable” tool 1 requirement is ambiguous (not automatable) TGDC Meeting, December 2011 Page 5

Findings: Requirement Interpretation Issues n We identified 10 requirement interpretation issues that require clarification n n The term “module” is often used in VVSG 1. 0 without reference to type (method/function, class, library) What constitutes “intentional exception” is not defined Logical nesting limit requirement needs clearer definition Scoping clarification needed for names that differ by only one character Module line count requirement (type of module, and which lines “count”) not defined (we assumed module=method) TGDC Meeting, December 2011 Page 6

Findings: Requirement Interpretation Issues n Additional requirement clarification needed for: n n n Indentation requirements not specific (“clear” and “consistent” is all the requirement specifies) Scoping clarification needed for “unique” name comparison in code Functional, test and branch statements not clearly defined Definition of what a “library module” is for Java is not specified Lowest shared access for a resource is not clearly defined TGDC Meeting, December 2011 Page 7

Findings: Even “Clear” Requirements May Be Interpreted Differently n n n An example: “Initializes every variable upon declaration where permitted” (VVSG 1. 0 2: 5. 4. 2) n int var. One=0; // VVSG conformant n static int var. Two; // non-VVSG-conformant What if Java automatically initializes var. Two to a value of 0 for you? In some cases, this happens at runtime Some manufacturers believe that the VVSG requirement is satisfied without explicitly assigning a value to a variable at declaration, and have written their code that way TGDC Meeting, December 2011 Page 8

Findings: Tool Automation Requires An Unambiguous Specification n Need mutual understanding of VVSG requirement meaning among voting system manufacturers, VSTLS and the EAC The “devil is in the details” to unambiguously specify requirements for source code We observed some “disconnects” between voting system code and SAMATE interpretation of VVSG requirements (from our July 2011 visit to Wyle Labs) TGDC Meeting, December 2011 Page 9

Findings: Where Possible, SAMATE Should Customize Tools Manufacturers are Already Using n Where voting system manufacturers are already using automated tools, SAMATE should (first) try to customize the tools that manufacturers are using n n n Two manufacturers currently using “Checkstyle” tool for Java code conformance verification SAMATE rules can complement the Checkstyle rules the manufacturer already uses without “adding another tool” This requires a dialog between SAMATE and manufacturers to identify which tools they are using TGDC Meeting, December 2011 Page 10

Findings: Need to Maximize SAMATE Resources n To make best use of our resources, SAMATE needs to know which tools, programming languages, compilers, libraries and coding conventions manufacturers are using n n 70% of voting system source code is written in C/C++, Java and C# languages (from discussion with VSTLs) Manufacturer’s compiler/library selection also dictate which tools SAMATE chooses for automation verification (e. g. GNU vs. Microsoft C++) If manufacturers are using industry coding conventions (not VVSG 1. 0 requirements), then SAMATE should focus its automated verification efforts against those industry coding conventions Synchronize SAMATE work with future VSTL testing work TGDC Meeting, December 2011 Page 11

Next Steps n TGDC review of customized Java tool and tests n n n Discuss any issues with the SAMATE work Resolve those issues Distribute SAMATE tool rules and tests to manufacturers and VSTLs n n n Two manufacturers are using the Checkstyle tool now (using their own custom rules) Get manufacturer/VSTL feedback on their interpretation and implementation of VVSG requirements in their tools Resolve those differences (via RFI or other means) Incorporate changes into future releases of VVSG as appropriate Publish a final distribution of customized tools and tests TGDC Meeting, December 2011 Page 12

Next Steps n Have a discussion with voting system manufacturers and VSTLs to determine where SAMATE should focus its future efforts in: n n Tools Programming languages and compilers Coding conventions Future VSTL work (new systems, re-certifications) TGDC Meeting, December 2011 Page 13

Next Steps n Investigate how SAMATE can contribute to automated conformance verification against VVSG 1. 1 software integrity requirements n n Most VVSG 1. 0 “style” requirements are removed in VVSG 1. 1 (in lieu of using industry-accepted coding style conventions) VVSG 1. 1 requirements are more “integrity focused” n n n Not trivial to find integrity bugs in source code Requires more sophisticated tools than “style checkers” Requirements include identifying race conditions, deadlocks, livelocks, pointer validation, dynamic memory allocation, numeric overflows and CPU traps TGDC Meeting, December 2011 Page 14

Next Steps n Integrate SAMATE VVSG work with SAMATE “Common Weakness Enumeration (CWE) Compatibility Effectiveness” program for testing automated vulnerability analysis tools n n n CWE is a unified, measurable set of software weaknesses defined in an online dictionary sponsored by DHS and implemented by MITRE Tools are assessed against their ability to identify CWEs in software with known weaknesses and known source code “complexities” (that can influence a tool’s ability to correctly identify a weakness) SAMATE can leverage this work in developing test suites to verify tool effectiveness against VVSG source code integrity requirements TGDC Meeting, December 2011 Page 15

Summary n SAMATE was successful at fully or partially automating Java source code verification against VVSG 1. 0 requirements that could be automated n n n Need to resolve semantic definitions of some VVSG 1. 0 software requirements (with manufacturers, labs and EAC) for uniform automated conformance verification uniformly across all tools Need to prioritize SAMATE resources against voting system manufacturer’s tools, languages, compilers and coding conventions to maximize impact for future work Need to look forward to VVSG 1. 1 software integrity requirements, and how SAMATE can assist manufacturers and VSTLs through tool effectiveness testing TGDC Meeting, December 2011 Page 16

Discussion TGDC Meeting, December 2011 Page 17