Introduction to design verification

12/17/15

Agenda

Why Verification?
Verification Alternatives
Languages for System Modeling and Verification
Concluding Remarks

12/17/15

Why Verification

Goal of verification:

Demonstrate functional correctness of a design

Attempt to find design errors
Attempt to show that design implements specification

Importance of Verification

Costs of design errors can be high

(think Pentium Floating-Point Error ~ $475M!)
According to [1], verification consumes about 70-80% design effor
t in current systems design

[1] J. Bergeron, Writing Testbenches: Functional Verification of HDL Models

mic Publishers, 2000

12/17/15

Kluwer Acade

Verification Reconvergence Model

Verification checks a transformation for correctness

RTL Design and Coding

Synthesis
Physical Design

Reconvergence Model:
Transformation
Initial
Specification

Transformation
Result
Transformation

12/17/15

Verification of RTL Design

The Idea:

RTL Coding

Written
Specification

RTL Code

Verification

How it works
RTL Coding
Written
Specification

Interpretation
RTL Code

Verification

12/17/15

RTL Synthesis flow

12/17/15

Verification in the design cycle

12/17/15

Trend of Verification Effort in the Desig

n

Verification portion of design increases to anywhere fr

om 50 to 80% of total development effort for the desi
gn.
Earlier

Now

Code

Verify (30 ~ 40%) Synthesis

Code

Verify (50 ~ 80%)

P&R

Synthesis

P&R

Verification methodology manual, 2008TransEDA

12/17/15

Percentage of Total Flaws

About 50% of flaws are functional flaws.

Need verification method to fix logical & functional flaws

From Mentor presentation material, 2003

12/17/15

Some more interesting stats ..

Another recent independent study showed that

mor

e than half of all chips require one or more re-spin

s, and that functional errors were found in 74% of t

hese re-spins.
With increasing chip complexity, this situation could
worsen.

Who can afford that with >= 1M

12/17/15

Dollar NRE cost?

10

Bug Fixing Cost in Time

Cost of fixing a bug/problem increases as design progr

esses.
Need verification method at early design stage

Cost of
Fixing
a Problem

12/17/15

Behaviora
l
Design

RTL
Design

Gate
Level
Design

Device
Production
11

Testbench: Basic building block

A testbench is HDL code to verify a module

Apply input vectors to module inputs

Check module outputs
Report errors to user

Why use a testbench ?

Portability - testbench will work on any HDL simulator

Automatic checking - don't have to interpret waveform
Expressability - can use the full semantics of HDL to:

12/17/15

generate input vectors (possibly from input file)

check output vectors
control simulation

12

Simple HDL Testbench

Testbench Module

Module Instance:
Device
Under
Verification
(DUV)

12/17/15

13

Comparing approaches

Visual inspection

Output comparison

Only practical for small designs

Automatic support: timing diagram editor
Effective when a good reference model is available
Used by ASIC foundries - Gold vectors are legal definition o
f a functional chip

Output checking

12/17/15

Most difficult to code

Mandatory for large designs

14

Testbench Approaches Visual Inspection

Testbench File

Stimulus
Generator

12/17/15

Device
under
Verification
(DUV)

Waveform Viewer
OR
Text Output

15

Testbench Approaches
Output Comparison
Testbench File

Gold
Vectors
Reference
Model
Output
Comparator

Stimulus
Generator

12/17/15

Device
under
Verification
(DUV)

16

Error/Status
Messages

Testbench Approaches
Self-Checking
Testbench File
Input Signals
Output
Signals
Stimulus
Generator

12/17/15

Device
under
Verification
(DUV)

Error/Status
Messages

Output
Checker

17

What a testbench is supposed to do?

Self-checking testbenches

12/17/15

Identify important features

Create conditions that test these features
Check conditions
Write message when error occurs
Insert errors to demonstrate when self-check fails
Test for varying values of all possible input values

18

Completion Metrics; How do we know wh

en the verification is done?

Emotionally or Intuitively;
Out of money? Exhausted?
Competing product is there.
Software people are happy with your hardware.
There have been no bugs reported for two weeks.

More rigorous criteria;

All tests passed
Functional Coverage
Code Coverage
Bug Rates have flattened toward bottom.

12/17/15

19

Agenda

Why Verification ?
Verification Alternatives

Simulation
Emulation
Prototyping
Formal verification
Semi-Formal verification

Languages for System Modeling and Verification

Concluding Remarks

12/17/15

20

Overview of Verification Methodologi

es

er
F a st

duct
o
r
p
inal
f
o
t
Prototyping
oser
l
c
,
d
e
e
sp
Hardware
Accelerated
Simulation

Emulation

Simulation

Basic
verification
tool

12/17/15

Semi-formal
Verification

Bigg
e

r cov

erag
e

Formal
Verification

21

Software Simulation

Dynamic verification method

Bugs are found by running the design implementatio
n.
Thoroughness depends on the test vector used.
Some parts are tested repeatedly while other parts ar
e not even tested.
Testbench

Other parts
are not even
tested.

DUV

a = 1;

Some part of
the design is
tested
repeatedly.

#20 b = 1;
$display (status is = %d,c);
...
12/17/15

22

Cycle-Based Simulation

Simulate the behavior of the design cycle-by-cycle.

Cycle-accurate information is provided as a result of
simulation.
Only signals at the flip-flop input are evaluated to be
stored, not internal signals of combinational logic.

Combinational
logic

12/17/15

Combinational
Combinational
logic
logic

23

Cycle-based vs. Event-driven

Cycle-based

Event-driven

Timing resolution

Clock cycle

User-defined minimum
delay

Evaluation time
point

Rising/falling/bot At the occurrence of

h clock edges
events

Evaluation node

Every flip-flop
boundary

At the output of every

logic gate on the
event propagation
path

Simulation time

Proportional to
the (number of
cycles) times
(C/L size *
number of F/Fs)

Proportional to the
number of events
(circuit size* no. of
cycles*event density)

12/17/15

24

Software Simulation

Pros
The design size is limited only by the computing re
source.
Simulation can be started as soon as the RTL descri
ption is finished.
Set-up cost is minimal.

Cons
Slow (~100 cycles/sec) ; Speed gap between
the speed of software simulation and real
silicon widens. (Simulation speed = size of
the circuit simulated / speed of the simulation
engine)
The designer does not exactly know how
12/17/15
25
much percentage of the design have been

Emulation

Imitating the function of another system to achieve th

e same results as the imitated system.
Usually, the emulation hardware comprises an array o
f FPGAs (or special-type processors) and interconnect
ion scheme among them.
About 1000 times faster than simulation.

Prototyping

Simulation
12/17/15

Hardware
Accelerated
Simulation

Emulation

26

Emulation

User logic design is mapped to emulation board with

multiple FPGAs or special processors.
The emulation board has external interconnection har
dware that emulates the pins of final chip.
Emulation hardware with multiple FPGAs

Logic design

Design
mapping

&
>
&

12/17/15

External pins
27

Prototyping

Pros
Higher (than emulation) clock rate (over 1M cycles
/sec) due to specific design of prototyping board.
Components as well as the wiring can be customiz
ed for the corresponding application.
Can be carried along. (Hardware Emulation? Forge
t it!)
Cons
Not flexible for design change
(Every new prototype requires a new board archite
cture. / Even a small change requires a new PCB.)

12/17/15

28

Overview of Verification Methodologi

es

Formal verification
Application of logical reasoning to the development of digita
l system
Both design and its specification are described by a language
in which semantics are based on mathematical rigor.

Semi-formal verification
Combination of simulation and formal verification.
Formal verification cannot fully cover large designs, and simu
lation can come to aid in verifying the large design.

Simulation

Semi-formal
Verification

Formal
Verification

More complete verification

12/17/15

29

Formal Verification

Objective

Pros

Check properties of model with all possible conditions

Assures 100% coverage.
Fast.

Cons

12/17/15

Works only for small-size finite state systems.

Uncomfortable due to culture difference (E.g., engineers are
not familiar with the use of temporal logic used for property
description in Model Checking)

30

Formal Verification : Equivalence Checker

Equivalence checker compares the golden model with the refined

model.
Golden
Golden
Model
Model

Refined
Refined
Model
Model

Functional representations are extracted from the designs and co

mpared mathematically.
Pros
Exhaustive design coverage
Very fast

Cons
Memory explosion

Tools such as LEC (Verplex), Formality (Synopsys), FormalPro (Me

ntor) supports Equivalence checking.

12/17/15

31

Semi-Formal Verification - Assertion

Assertion-based verification (ABV)

Assertion is a statement on the intended behavior of a desi

gn.
The purpose of assertion is to ensure consistency between th
e designers intention and the implementation.

Key features of assertions

12/17/15

1. Error detection : If the assertion is violated, it is detected b

y the simulator.
2. Error isolation : The signals related to the violated assertio
n are identified.
3. Error notification : The source of error is reported to the u
ser.

32

Semi-Formal Verification - Assertion

Example of assertion-based bug detection

PCI
PCIDMA
DMAController
Controller

Identify signals related to

the violated assertion

PCI

devsel
assertion is
violated!

event devsel :
if (FRAME=0) [1..4]
(DEVSEL=0)
assert(devsel);

Report to the
user!!
12/17/15

33

Semi-Formal Verification - Assertion

Simulation Quality of assertion-based verification
Number of bugs found

Simulation with assertions

Formal verification

Setup
testbench
12/17/15

Describe
assertions

Efficiency of
assertion

Simulation

Time, Effort
By IBM in Computer-Aided Verification 2000
34

Semi-Formal Verification - Coverage

Coverage-directed verification
Increase the probability of bug detection by checking the qual

ity of stimulus
Used as a guide for the generation of input stimulus
Test
TestPlan
Plan
(Coverage
(Coverage
Definition)
Definition)

Directives
Directives

Coverage
Coverage
Reports
Reports

12/17/15

Random
Random
Test
Test
Generator
Generator

Coverage
Coverage
analysis
analysis

Test
TestVectors
Vectors

Simulation
Simulation

35

Semi-Formal Verification - Coverage

Coverage metrics for coverage-directed verification

Code-based metrics
Line/code block coverage
Branch/conditional coverage
Path coverage
Circuit structure based metrics
Toggle coverage
Register activity

State-space based metrics

Pair-arcs : usually covered by Line + condition coverage
Functional coverage metrics

12/17/15

% of specification items satisfied

36

Semi-Formal Verification - Coverage

Coverage Checking tools

VeriCover (Veritools)
SureCov (Verisity)
Coverscan (Cadence)
HDLScore, VeriCov (Summit Design)
HDLCover, VeriSure (TransEDA)
Polaris (Synopsys)
Covermeter (Synopsys)

12/17/15

37

Semi-Formal Verification

Pros

Cons

Designer can measure the coverage of the test environment

as the formal properties are checked during simulation.
The simulation speed is degraded as the properties are check
ed during simulation.

Challenges

12/17/15

There is no unified testbench description method.

It is difficult to guide the direction of test vectors to increase
the coverage of the design.
Development of more efficient coverage metric to represent
the behavior of the design.

38

Speed Comparison
Speed (Cycles/sec, log scale)

10MHz

1~10MHz
500KHz

1MHz

100kHz

100 kHz

10 kHz

50-70Hz

100Hz
0 kHz

12/17/15

Software HardwareHardware
Simulation Accelerated
emulation
Simulation
(from Quickturn
(from
presentation)
Quickturn/Dynalith
Presentation)

Prototyping

39

Semi-formal
(Assertionbased
verification)

Design Complexity
Gate counts

Comments

Simulation/Semiformal verification

Unlimited

Emulation/Hardwar
e-accelerated
simulation

1M~16M gates

Depends on the
number of FPGAs in
the architecture

Prototyping

1M~5M gates

Depends on the
components on the
board

Formal verification

< 10K gates

Limited to about 500

flip-flops due to state
explosion

12/17/15

40

Verification Time vs. Coverage

Coverage

Semi-formal
Emulation
/Accelerated simulation
Prototyping

Simulation

Redirection
of
testbench
constraints

Simulatio
12/17/15
n setup

Semi-formalEmulation
setup
setup

Prototypin
g setup

Verification Time

41

Agenda

Why Verification ?
Verification Alternatives
Languages for System Modeling and Verification

System modeling languages

Testbench automation & Assertion languages

Concluding Remarks

12/17/15

42

Language Heritage for SoC Design

New languages are developed to fill the productivity g

ap.
JAVA

Language for
Software development

Assembly
Language for
Hardware test

C++
C

SystemC

TestBuilder
Vera

Language for
Hardware description

Schematic
12/17/15

Past

SystemVerilog
Verilog
VHDL
present

43

Future

System Description Languages Summar

y
Languge

Pros

C/C++

HDL
(Verilog,
VHDL)

Familiarity

SystemC

Easily

SystemVerilo
g
12/17/15

Easy to write test

vectors/environmen
t

Easy

to describe H/W
designs

connected to
C/C++ codes.
Easy to model system
behaviors.
Easy

to learn for the

HDL designers.
Easy to model system
behaviors.

Cons
Unable

to handle some
hardware
environments.

Focuses

on the lowerlevel designs.

Improper for system
modeling.
Limited

tools
(simulation, synthesis,
etc.)

Few

tools (simulation,
synthesis, etc.)
Subset support
44

ASIC Verification Methods

Running Speed
100MHz

Ideal Verification
10MHz
Solution
1MHz

100KHz

Make it cheaper

Make it faster

Real Silicon

Rapid Prototype
HW Emulator

10KHz
1KHz
100Hz
10Hz
12/17/15

HW Accelerator
SW Simulator

Investment
45

Overall Functional Verification Flow

Architecture Define
Microcode
Description
Microcode
Verifier

RTL Description
(Verilog HDL)
Synthesis
Gate Level
Simulation

Hardware
Emulation

12/17/15

Verification Completed

RTL Simulation

For version
control

46

Concurrent Verification
Without Emulation
SW
HW

Sequential
Verification

Design
Design

CHIP

System
Integration

Code
Hardware
Integration

Build
Design

Fab

Debug

Debug

Back
annotation

With Emulation
SW
HW

Design

12/17/15

Concurrent
Verification
Code

Design

Build

CHIP
HW emulation

Time

Design
Chip
Debug

Fab
HW integration
& HW Debug
Sys integration
& SW Debug

Final
Integration

Back annotation

Early to
Market!!
Debug

47

Debug

Agenda

Why Verification ?
Verification Alternatives
Languages for System Modeling and Verification
Concluding Remarks

12/17/15

48

Concluding Remarks

Verification is challenging; It needs strategy!

Strategy is to apply each method when appropriate.
Verify as early as possible; Catch the bug when it is small and still
isolated in a smaller region. (Dont wait until it grows and kills yo
u.)
1st step: Apply formal methods
Static formal verification
Assertion-based verification

2nd step: Coverage driven verification aids in bringing closure

Code and functional coverage , if not covered design doesnt work !!
Selecting the proper verification methodology

3rd step: Emulate design

Emulate IP operation in FPGA

12/17/15

49

Concluding Remarks

Powerful debugging features handling both hardware part and s

oftware part are required.
Language, Tool/Data Interfaces need standardization.
DFV (Design for Verification) ; You lose in the beginning, but will
win later, like Design for Reuse.

12/17/15

50

Thank You!!

12/17/15

51

Questions ???

12/17/15

52

Appendices

12/17/15

53

Debugging Design in the FPGA

Embed logic analyzer with user design in EDIF format

Logic to store pre-registered signals into the probing memory.
Logic for trigger condition generation.
Triggering condition is dynamically configured.

Internal node extraction

Sometimes the designer wants to watch internal nodes in the

design.
Internal node probing Top block
enables this by
wiring-out the internal Sub-block
nodes to the boundary
of the DUT top block.

DUT

Built-In
Logic
Analyzer

Internal node
12/17/15

BILA, Dynalith Systems

54

External
Dump
Memory

RTL Debugging Feature

Emulation is based on gate-level netlist.

Gate-level netlist generated from the synthesis tools h
as too complex name styles difficult to trace manually.
Techniques to resolve RTL symbol names from the gat
e-level symbol names and to provide debugging enviro
nment in RTL name spaces is required.
Insert RTL instrumentation IP for debugging
Design flow
Read RTL design (Verilog, VHDL)
Generate instrumented RTL design (spiced with triggering and d
ump logic)
Synthesis
Compile (mapping & PAR)
DiaLite (Temento), Identify (Synplicity)

12/17/15

55

RTL Debugging Feature

Instrumentation IPs for debugging logic blocks mappe

d into FPGAs.

Trigger
Logic Equation Module
History Register
Transaction Register
Random Generator
Traffic Analyzer

Instrumentation IPs are

interconnected to
support various
configurations.

12/17/15

Structures of the RTL design

Interconnection of instrumentation IPs

DiaLite from Temento

56

Connecting Actual Chip to the Simulator

Building a correct and fast reference model for the hardware is v

ery difficult.
Use the actual discrete chip
for the IP (or FPGA).

Control the clock signal to the

actual chip (or FPGA), i.e,
slow down and
synchronize with the HW simulator
and SW debugger
in the host machine.

Application
FPGA prototyping
HW/SW co-verification
Silicon validation

12/17/15

57

from SimPOD

Microprocessor Design Verification Methodolo

gy
more refined model

CPU
C Language
Instruction
Behavior
In C
(Polaris)
Virtual Chip

HDL

Microarchitecture
in C

FlexPC

RT-Level
in Verilog

Gate-Level
in Verilog

Using

MCV

PLI

Real
Mother-board
H/W

Peripherals

12/17/15

Virtual PC in C language
(VPC)

MCV : Microcode Verifier

58
PLI : Programming Language
Interface

Prototyping

Special (more dedicated and customized) hardware a

rchitecture made to fit a specific application.

Prototyping

Simulation

12/17/15

Hardware
Accelerated
Simulation

Emulation

59

A Prototyping Example
Switch board

Prototype of 4-Port G
igabit Ethernet Switc
h

Two Xilinx Virtex-E 20

00 FPGAs are on FPG
A board.
Four FPGA boards are
used.
Xilinx FPGA
Processor board cont
ains PCI bridge and M
PC860 microprocesso
PCI bridge
r.

Courtesy of Paion, Inc.

12/17/15

MPC860
microprocessor

Processor board

60