Heuristic Stimuli Generation For Coverage Closure Exploiting Simulation Feedback

Giovanni Squillero
Politecnico di Torino - Italy CAD Group ( )
giovanni.squillero@polito.it

GOAL
To propose a methodology for coveragedirected stimuli generation based on simulation feedback Such stimuli could be added as new content to improve existing validation suites

DVClub 2010-01-18

giovanni.squillero@polito.it

Acknowledgements
Danilo Ravotto Ernesto Sanchez Matteo Sonza Reorda Alberto Tonda

+ many others

DVClub 2010-01-18

giovanni.squillero@polito.it

Outline
Proposed methodology Case studies Conclusions

DVClub 2010-01-18

giovanni.squillero@polito.it

Design Choices
Being able to tackle real problems Develop a versatile and broadly applicable methodology
Compatible with different environment Compatible with any coverage metric

Minimize effort to change goal/target

Exploit common aspects

DVClub 2010-01-18

giovanni.squillero@polito.it

Feedback-Based Approach
Simulation-based approach Exploits feedback from simulation Incremental improvement/refinement of the solution (trial-and-error) Trade-off between computational resources and confidence May exploit heuristics or problem-specific knowledge
DVClub 2010-01-18 giovanni.squillero@polito.it 6

Proposed Methodology
Stimuli

Stimuli Generator

System

Feedback
DVClub 2010-01-18 giovanni.squillero@polito.it 7

Stimuli
Stimuli

Stimuli Generator

System

Feedback
DVClub 2010-01-18 giovanni.squillero@polito.it 8

Stimuli
Sequences of bits Sequences of keys Full fledged assembly language programs VHDL test case External world

DVClub 2010-01-18

giovanni.squillero@polito.it

Stimuli Generator
Stimuli

Stimuli Generator

System

Feedback
DVClub 2010-01-18 giovanni.squillero@polito.it 10

Stimuli Generator
Exploit an Evolutionary Algorithm to generate stimuli to maximize a given function

DVClub 2010-01-18

giovanni.squillero@polito.it

11

Evolutionary Algorithms
Meta-heuristic optimization algorithm based on the concept of population and exploiting some principles of natural evolution

DVClub 2010-01-18

giovanni.squillero@polito.it

12

Evolutionary Algorithms
Succession of random and deterministic steps
A systematic way of throwing dices Better than pure random
The great effect produced by the accumulation in one direction, during successive generations, of differences absolutely inappreciable by an uneducated eye

DVClub 2010-01-18

giovanni.squillero@polito.it

13

Evolutionary Algorithms
Population
Multiple solutions considered in each step More resistant than pure hill-climbing Different solutions may interbreed

DVClub 2010-01-18

giovanni.squillero@polito.it

14

Evolutionary Algorithms
Why using an evolutionary algorithm?
Adaptative Able to find unexpected solutions Better than random

DVClub 2010-01-18

giovanni.squillero@polito.it

15

Evolutionary Algorithms
Problem: Fitness function
GOAL: Optimize the wheel FITNESS: Minimize the number of bumps

DVClub 2010-01-18

giovanni.squillero@polito.it

16

Evolutionary Algorithms
Problem: Black magic

DVClub 2010-01-18

giovanni.squillero@polito.it

17

GP (MicroGP)
CAD Group general-purpose evolver
3 versions (only 2 released under GPL) Project started in 2002 11 developers + contractors, students,

Current version
300 file, > 40,000 lines in C++

DVClub 2010-01-18

giovanni.squillero@polito.it

18

GP (MicroGP)
Evolutionary Optimization: the GP toolkit
E. Sanchez, M. Schillaci, G.Squillero Springer, 2010
ISBN: 978-0-387-09425-0

DVClub 2010-01-18

giovanni.squillero@polito.it

19

GP (MicroGP)
http://ugp3.sourceforge.net/
MicroGP++ (aka. ugp3, GP3) Information Download Credits

DVClub 2010-01-18

giovanni.squillero@polito.it

20

System & Feedback

Stimuli

Stimuli Generator

System

Feedback
DVClub 2010-01-18 giovanni.squillero@polito.it 21

System
Strongly problem dependant Model via simulation/emulation
HDL (netlist to high-level) HW accelerated (e.g., exploiting FPGA) Architectural simulator ISA simulator

Real device

DVClub 2010-01-18

giovanni.squillero@polito.it

22

Feedback (examples)
From simulation
Code coverage metrics (e.g., instruction coverage) HW specific metrics (e.g., toggle coverage) High-level information (e.g., FSM coverage)

From the real system

Performance counters Physical measures (e.g., temperature, time, power consumption)
DVClub 2010-01-18 giovanni.squillero@polito.it 23

Outline
Proposed methodology Case studies Conclusions

DVClub 2010-01-18

giovanni.squillero@polito.it

24

Design Verification
Devise a set of programs maximizing different coverage metrics

DVClub 2010-01-18

giovanni.squillero@polito.it

25

Feedback
Code coverage metrics
Statement coverage (SC) Branch coverage (BC) Condition coverage CC) Expression coverage (EC)

HW specific metric
Toggle coverage (TC)

DVClub 2010-01-18

giovanni.squillero@polito.it

26

System
DLX/pII
Cleaned and simplified MIPS intended primarily for teaching purposes 32-bit load/store architecture, 5-stage pipeline VHDL RTL description
4,558 statements 3,695 branches 193 conditional statements (1,764 expressions) 8,283 logic bits
giovanni.squillero@polito.it 27

DVClub 2010-01-18

Experimental Results

DVClub 2010-01-18

giovanni.squillero@polito.it

28

Experimental Results
2,978 instructions

230K instructions

DVClub 2010-01-18

giovanni.squillero@polito.it

29

Experimental Results
20h 33h 27h 37h 95h

1 week

DVClub 2010-01-18

giovanni.squillero@polito.it

30

Post-silicon Verification
Generate functional test programs for postsilicon verification Activity performed in collaboration with the ETM Group, Intel (Phoenix) Intel Pentium 4
42-55 millions of transistors 2GHz clock NetBurst architecture
DVClub 2010-01-18 giovanni.squillero@polito.it 31

System & Feedback

Real Pentium 4 (no simulation) Performance counters as metric
Introduced in 1993 in IA32 architecture 48 event detectors + 18 programmable counters Instruction-tagging (for discriminating nonspeculative performance events)

DVClub 2010-01-18

giovanni.squillero@polito.it

32

Experimental Results
Two sets of experiments:
Mispredicted branches over the total branches Clock cycles in which the trace cache is delivering ops to the execution unit instead of decoding or building traces (hard metric!)

Intrinsic features of the architectural design Time (both): 12h/program

Haifa 2008

G. Squillero

33

Mobile Phone
12m-activity performed in collaboration with Motorola Research Center in Turin (Automation of Current Drain Measures)

DVClub 2010-01-18

giovanni.squillero@polito.it

34

System
Real prototypical mobile phone
P2K platform GSM/3G networks Complex applications

Radio Communication Analyzer, anechoic chamber Controllable power supply Custom hardware
DVClub 2010-01-18 giovanni.squillero@polito.it 35

Stimuli
Sequence of keys Power supply Network status

DVClub 2010-01-18

giovanni.squillero@polito.it

36

Feedback
Physical measures
Power consumption (+ autocorrelation) Network status

FSM transition coverage

DVClub 2010-01-18

giovanni.squillero@polito.it

37

Experimental Results
Major bugs discovered in the prototype
Test infrastructure LCD display Fake deep-sleep state

DVClub 2010-01-18

giovanni.squillero@polito.it

38

Outline
Proposed methodology Case studies Conclusions

DVClub 2010-01-18

giovanni.squillero@polito.it

39

Conclusions
Simulation-based & feedback-based Evolutionary algorithm Broadly applicable Human resources
Limited (set-up )

Computational resources
Easily parallelizable (generation level) Trade-off quality vs. effort
DVClub 2010-01-18 giovanni.squillero@polito.it 40

Conclusions
May exploit other methodologies
Useful starting point Effective completion

May be coupled with other methodologies

Rule-based instruction randomizers Simulation-based approaches

DVClub 2010-01-18

giovanni.squillero@polito.it

41