Using Formal Specifications to Monitor and Guide Simulation

Using Formal Specifications to Monitor and Guide Simulation: Verifying the Cache Coherence Engine of the Alpha 21364 Microprocessor Serdar Tasiran Systems Research Center, HP Labs (formerly Compaq) Yuan Yu (Microsoft Research, formerly Compaq) Brannon Batson, Scott Kreider, (Intel, formerly Compaq)

The Problem l Given n n l Verify that n l A complex protocol specified formally A hardware implementation All executions of implementation are consistent with protocol spec An “implementation verification” problem n Properties of protocol verified separately

Alpha 21364 (EV 7) System Block Diagram 0 4 8 n n M IO 1 5 9 M IO 2 6 10 M IO Distributed shared memory Up to 256 processors, 32 GB per processor 3 7 11 M IO

EV 7 Cache Coherence: l Cache coherence protocol spec (~3 K lines of TLA, written by architects and us) n n Executable Not list of properties l Implementation in hardware (~20 K lines of HDL code) R EV 7 C system data buffers SVDB, FB 0, FB 1 Z 0 mem l Z 1 mem Does the hardware implement the protocol spec correctly? core & cache

Why is the problem difficult? l Thousands of state variables per processor n Parallelism, deep pipelining, speculative execution, redundancy, . . . l Need 4+ processors for an interesting system l Out of reach of automatic formal methods n Limited to several hundred state variables l Decomposition methods difficult for non-specialists, large design teams l Complete verification of hardware against protocol not practical l Simulation only viable approach n Even simulation is expensive

Validation Guided by Formal Spec Coverage Automated input generation Simulation Coverage analysis on formal spec Correctness checking using formal spec

Validation Guided by Formal Spec Coverage “Automated” input generation Simulation 1 Coverage analysis on formal spec Correctness checking using formal spec

Contributions: 1: Formal Spec + Model Checker as Monitor l Spec written in formal language n Properties of spec can be verified formally n Model checker checks properties satisfied during simulation: u u l More reliable than hand-written code More flexible than automatically generated assertions Must relate Implementation state Spec state n Devised two-phase mapping approach n Applicable to complex designs by non-specialists

Validation Guided by Formal Spec Coverage “Automated” input generation Simulation 2 Coverage analysis on formal spec Correctness checking using formal spec

Contributions 2: Coverage analysis and input generation using formal protocol spec l Formal spec encapsulates design intent n n l Full coverage = All scenarios exercised Spec at same level of abstraction as existing coverage data Model checker used to n Measure coverage, detect gaps n Generate simulation input traces to reach coverage holes n Determine if unexercised scenario actually possible

Outline l l The cache-coherence protocol The EV 7 cache-coherence engine Spec-guided simulation Conclusion

The EV 7 Coherence Protocol l Distributed shared memory n l Directory-based protocol n n l l l Each address belongs to a “home node” but may be in other caches M 0 M 1 IO IO M 4 IO IO IO M 10 IO M 7 IO M 9 IO M 6 IO M M 3 IO M 5 8 M 2 M 11 IO IO Cache states: Modified (Dirty), Exclusive (Clean), Shared, Invalid Directory states: Local, Shared, Exclusive, Incoherent Directory distributed with memory at each node CPU requests that miss in local caches are sent to home node Home node may forward request to other nodes n Directory In Flight Table (DIFT) keeps track of pending requests

Example: write, remote sharers Read. Mod Shared. Inv R H Blk. Exclusive S S S Inval. Ack l Conditions: n l home is remote, directory state is shared Actions: n n read-exclusive request to home sends invalidation requests to sharers, sends data back to requester with invalidation count (early exclusive reply) sharing nodes reply to requester with invalidation acknowledgements requester proceeds when data arrives, but must “stall” incoming requests and potential writeback of line until all Inval. Acks are received

TLA Description for Protocol l Temporal Logic of Actions [Leslie Lamport] Formal language for writing high-level specs of concurrent, reactive systems Very expressive. Incorporates n n l First-order logic, set theory, temporal operators Sets, queues, records, tuples, … EV 7 protocol description is a TLA formula

One Protocol Action Read. Mod R Shared. Inv H Blk. Exclusive. Cnt S S S Preconditions Messages sent and state variables updated

Outline l l The cache-coherence protocol The EV 7 cache-coherence engine Spec-guided simulation Conclusion

Alpha 21364 Chip Block Diagram Data Buffers L 2 Data Array L 2 Tag Array L 2 Cache Controller Core L 1 Cache Router IPx 4 I/O Memory RDRAM Controller 0 Memory RDRAM Controller 1 Data Address & Control

Directory in Flight Table (DIFT) l l Front end of memory controller Tracks up to 32 in-flight transactions Cache State Lookup (from cache controller) DIFT Request Coherence Engine Directory State from Memory New directory to Memory Forwards and responses to other CPUs

DIFT block diagram From CBox From Back End From CBox New Request Decode zc_dft_acc zc_dft_ros Address File (zx_dft_af*) addr maf pid vdba vdbv fb directory Event File cmd src ack vic lpr Proto File rd fwd rsp inv wr vcp akp To RBox Messages to the Ring Address Output Logic (ao*) Issue Logic (zx_dft_isp*) New Directory Output (aod) Grant logic to everywhere Rd/Wr Requests to the Zbox middle New Directory Logic (zx_dft_plt) To Back End Protocol Logic (zx_dft_plc) Next DIFT State DIFT Free List DIFT Conflict Array

Outline l l The cache-coherence protocol The EV 7 cache-coherence engine Spec-guided simulation Conclusion

Validation Guided by Formal Spec Coverage “Automated” input generation Simulation 1 Coverage analysis on formal spec Correctness checking using formal spec

Formal Spec as Simulation Monitor Spec State-Space fabs Implementation State-Space Model checker (TLC) checks if transition is legal fabs : Abstraction mapping

Refinement Mapping Issues l Protocol transactions look instantaneous at spec level but in the implementation they happen n n over many clock cycles, interleaved with other actions Actioni 1 (pj 1, ak 1) Actioni 2 (pj 2, ak 2) Actioni 3 (pj 3, ak 3) time l Want designers, developers to write the mapping l Implementation has parallelism, pipelining, speculative execution, redundancy AND many processors l Burch, Dill style “flushing” not practical

The Refinement Mapping Preconditions Messages sent and state variables updated Actioni 1 (pj 1, ak 1) Actioni 2 (pj 2, ak 2) Actioni 3 (pj 3, ak 3) time

A two-phase recipe for refinement mappings Collect “tokens”, e. g. , cache state looked-up, invalidate message sent, directory state written 1. Implementation state Intermediate state l Determine n interfaces: write, read ports n state machines that relate to protocol state l Watch n Messages crossing interfaces n Updates to state machines l Record in intermediate state l IO R EV 7 system data buffers C Z 0 core & cache Z 1 mem Record in intermediate state for processor pj 3 address ak 3 Actioni 3 (pj 3, ak 3) time

A two-phase recipe for refinement mappings l All tokens collected. “Fire” action. 2. Intermediate state Protocol spec state n For each protocol transaction, check when u u n All preconditions hold All state updates happen, all required messages sent Update abstract state Preconditions and implementation state updates related to Actioni 3 (pj 3, ak 3) Abstract state gets updated here Actioni 3 (pj 3, ak 3) time

A two-phase recipe for refinement mappings 1. Implementation state Intermediate state n Hardware signal transitions Protocol events (tokens) 2. Intermediate state Protocol spec state n Protocol event sequences Protocol transactions (actions) l l Component implementers can write step 1 System architects can put together step 2 Distinguishes protocol errors from component implementation errors Well-defined, clean interface n Easier to keep implementation and spec consistent throughout design process n Modular description makes reasoning easier

Validation Guided by Formal Spec Coverage “Automated” input generation Simulation 2 Coverage analysis on formal spec Correctness checking using formal spec

Formal Spec as Coverage Model Spec State-Space fabs Implementation State-Space Model checker (TLC) records visited states fabs : Abstraction mapping

Model Checker Tracks and Improves Coverage Spec State-Space Path generated by model checker l l l Identify parts of spec not exercised by simulation Path in spec state space = unexamined scenario Problems: n n l Non-covered state Spec has too many states Not feasible to track coverage, generate paths for each Want to explore “qualitatively distinct” scenarios

Coverage Metric Defined on Spec States Coverage State-Space c 1 c 0 c 6 c 4 c 7 c 2 Spec State-Space c 3 c 5 c 8

Coverage Metric Examples l All possible directory state transitions: n l All legal combinations of n n l Invalid Exclusive Shared. Mask Request type Source of request (Local or remote) Cache state Directory state All possible transitions of some protocol state field in the DIFT n Waiting. For. Ack Waiting. For. Victim, …

Conclusions l Novel approach uses formal spec and model checker n to monitor simulation n to identify coverage gaps n to guide input generation l Found valuable by architects and verification engineers l EV 7 verification engineers want to use model checker to analyze their coverage data l EV 8 design started with formal specification first! l First attempt at verifying industrial implementation of this scale and complexity against formal spec

Actioni 1 (pj 1, ak 1) Actioni 2 (pj 2, ak 2) Actioni 3 (pj 3, ak 3) time Preconditions and implementation state updates related to Actioni 3 (pj 3, ak 3) Abstract state gets updated here Actioni 3 (pj 3, ak 3) time

Read. Mod R Shared. Inv H Blk. Exclusive. Cnt S S S

M 0 M 1 M 2 M 3 IO IO M M 4 5 6 7 IO IO M M 8 9 IO 10 IO 11 IO IO