Paper Review Hole Analysis For Functional Coverage Data

Paper Review: Hole Analysis For Functional Coverage Data by Oded Lachish, Eitan Marcus, Shmuel Ur, Avi Ziv @IL. IBM 0 -32069023 אבשלום אלידע 3 -10521372 לאוניד פרואנצ'נקו CAD of VLSI Systems (046880) 21/01/03

Functional Coverage • Functional Coverage -- A means of: – Measuring the efficiency of Functional Verification. – Finding how and in what directions to improve Test Generation in order to increase coverage. – And as a result, reach more corner verification cases and CATCH MORE BUGS !!! • Compare to older, less effective coverage methods (toggle, line, cross-state), Functional Coverage models closely the design functionality.

Functional Coverage: Motivation • Just testing is not enough: – Manual testing (MT) time-consuming and only gives us relatively simple cases. – Efficient bug-finding with random testing (RT), but RT machine-resource consuming and needs to be tracked closely for efficiency with a dedicated tool … • The 1 M$ question of verification: what level of coverage are we achieving? ! Without dedicated tool, cannot answer this.

Functional Coverage in a Nutshell • Define list (space) of complete functionality of design. Comprises of verification tasks i. e, cases we expect to reach. • Track testing results (mostly RT, also MT) and check which tasks were reached. (Groups of) tasks not reached are referred to as “holes”. • In order to “plug” the holes: – Change/improve test-generator or testing-environment as needed or, – Define restriction on coverage model if hole is not real case.

Functional Coverage – Cntd. • An Iterative Process: Resume testing to check improvement in total tasks covered report coverage again • Cases-reached / Total-Case-Space = % Coverage

Functional Coverage: An Iterative Process

Example Coverage Model: A Network Switch Receive Port • Define attribute groups. Rx-port consists of pairs. • Define attributes, for each attr. its possible values:

Receive Port – Packet Received Attribute Values Pkt Type Ethernet, IP-over-Eth, Layer 4 -o-IP-o-Eth, Eth. Control, … Pkt Length too short(0 -63 byte), small(64 -127), medium(128 -511), large(512 -1023), extralarge(1024 -1534), too long(1535 byte and up) Layer 2 Cast unicast, multicast, broadcast, unknown (destination only or source too) CRC in intact, wrong, missing?

Receive Port – Decision Attribute Values Forward to reject, own device, other device, CPU, multicast/broadcast reject reason accept, length: too short/long, CRC, collision, rx error … CRC out pass as received, append, recalculate

Hole Analysis • Concise presentation of coverage data crucial for effective analysis of coverage results. – Mere list of tasks not reached – all but meaningless! • Group together multiple single-element holes, Projected Holes: n-dimensional holes. More significance and clarity than mere list of uncovered tasks. – Case space: has N=7 dimensions. – 0 -dimensional hole: one point in case-space not reached: didn’t happen. – n-dimensional hole: no event reached for any of the cases where N-n+1 attributes were at a fixed value: hole of n=5, no IP pkts were rejected for any reason! – Group together 0 -dim. holes to more significant n-dim. holes.

Hole Analysis: Coverage Data Presentation • Aggregated Holes • Group holes together based on Hamming Distance = 1. (Hamming dist. = # of different attributes). Example with 2 attributes (1 -covered, 0 -uncovered) Accept Rej on Bad CRC Rej on Length own dev Rej collision port 0 0 0 1 1 port 1 1 1 0 0 port 2 1 1 0 0 Other dev 1 0 1 1 CPU 1 1 better: Partitioned Holes.

Conclusions • Coverage emerging as critical tool in verif. flow: • Close tracking of testing environment effectiveness, closely linked to RT. • Assessment (in %) of degree of above effectiveness, gives a feel of how much more verif. needs to be done. • Considering ever-increasing size&complexity of today’s industry designs, keeping verif. up to the task without diverse&informative coverage tool – next to impossible. • Diverse&informative: in-depth Hole-Analysis, smart grouping of holes to “serve” for the verif. engineer.

Food for Thought • Modeling of functional coverage closely coupled to many subjects studied in the course: • Data structures use BDD for compact representation of sets attributes. • Checking whether a task was covered satisfiability. • Coverage and Formal-Verification: Two Sisters • Formal: state a rule check for it’s tautology. • Coverage: state a rule in from of “task didn’t happen” check that it is unsatisfied. *) No mumbo-jumbo clip-art or meaningless graphics were used in the making of this paper review. Technion, Israel, January 2003. A. E, L. F.