Sie sind auf Seite 1von 18

Metrics in SoC Verification

What Can Metrics Tell Us?

Andreas Meyer
Verification Architect
academy@mentor.com
www.verificationacademy.com

WHAT CAN METRICS TELL US?


We are interested in a broad set of metrics giving
us insight into the entire verification flow:
The build, simulation and regression processes
Various aspects of the overall project

Yet, metrics must be actionable


Otherwise the process of measuring
and storing metrics data waste
project resources

Example of Various Verification Metrics


Process and
Focused Areas

Process and Focused Area Attributes and the Information Associated Metrics Can Provide

Design

Abstraction
level

Simulated performance

Stimulus

Source of
stimulus

Type of stimulus (CR,


firmware, graph,
legacy, etc.)

Checking

Build

Source of
checkers
Interface
activity
Categories of
coverage
Source and rev
of files

Run

Simulator/Emul
ator

Debugging

Area of failure

Regress

Which
simulations
Open bugs

IP
Coverage

Bug Status

Results of checkers

List of instantiated
blocks (and versions)

Checker abstraction
levels

Key internal states


RTL/stimulus/checker
reference model
Initial configuration
used
Host machine info
(memory, disk image
distance, etc.)
Commonality of cases
where many tests
report same failure

Abstraction level of
coverage

Simulation performance

Revision of tools

Errors found

Errors re-found (i.e.,


wasted simulation)

Bug discovery info:


stimulus, abstraction
level, checker

Metrics used to isolate


bug

Improvements in
coverage results
Bug closure information
(sim time, engineer
time, number of runs)

Correlating multiple metrics


Only when multiple metrics are correlated during analysis
does real value emerge
For example, for newly built revision of a system we might
want to answer the question:
What coverage was hit for a specific type of applied
stimulusat a particular level of abstraction of the
designand for a particular revision of the firmware?

Correlating multiple metrics gives us clearer view of the


circumstances that allowed us to hit specific coverage
item

Metrics as part of the build process


In large SoC environments, with significant code
churn, metrics can provide us insight into the
build process
For example, we might be interested in knowing:

Which IP blocks were used during the build process?


Where did each IP block originated?
Which version number was associated with each IP?
What level of abstraction was used for the build?

Event-based and non-event-based metrics


Metrics need not count multiple events to be useful
Metrics for the non-event-based build can be used to
qualify queries around specific code
For instance, we could determine the coverage
metrics associated with a specific IP version, or
identify what configuration or randomization that
was done during the build

Correlating event-based with non-event-based


metrics may be useful in checking completeness of
overall verification

Metrics as part of the simulation process


Metrics as part of simulation enable us to determine:
What happened during the simulation process
Which pieces of the simulation environment were used
How the pieces played together

Aspects of the simulation process that can benefit from


metrics include
Stimulus sources
Checking methods
Coverage metrics
Domain-specific performance
Simulation performance
Simulation configuration

Metrics and stimulus sources


Large SoC designs are likely to use a number of stimulus
sources within a single simulation
Metrics can help us
Measure which sources were used and help identify the type and
frequency of traffic generated by each source
Understand how the system was tested
Measure the productivity of various stimulus methods

Metrics and checking methods


Large SoC designs are also likely to use a number of
different checking methods
Metrics can help us
Ensure that the desired checkers are in place and are
receiving traffic
Identify the number and types of checks performed

Checking metrics, when correlated with other metrics,


provide a deeper understanding of the simulation
environments effectiveness and productivity

Metrics and coverage metrics


Coverage metrics are useful for identifying stimulus holes
Yet, cannot answer the questions, Did a specific test
activate and then propagate and event to a specific checker?
Mem
Operation

TLB
Tables

Process
or
Core

Cache
State

Cache
State

Coherent
Cache

TLB
Tables

Coherent
Cache

IP Core

Mem
Operation

MemoryIntensive
IP

IP Core

Fabric
Ops
Network Switch

RAM

Open
Pages
RAM Op
Queues

Mem
Access

Metrics and domain-specific performance


Understanding performance characteristics can be critical
Metrics can be used to measure domain-specific
performance and then used to calculate system-performance
Mem
Operation

TLB
Tables

Process
or
Core

Cache
State

Cache
State

Coherent
Cache

TLB
Tables

Coherent
Cache

IP Core

Mem
Operation

MemoryIntensive
IP

IP Core

Fabric
Ops
Network Switch

RAM

Open
Pages
RAM Op
Queues

Mem
Access

Metrics and simulation performance


Metrics can help us track performance and provide an
early detection mechanism to identify issues

By correlating performance with code revisions, it may be


possible to link performance degradations to the
introduction of specific blocks of code

Metrics and simulation configuration


Each simulation may have run-specific configurations that
can affect other aspects of the verification process
Probably the most obvious configuration is the random seed
that was generated for each run

That seed or other parameters may be used to select tests or


change the configuration of the simulation

Reporting each configuration option as a part of metrics


allows configuration changes to be correlated with other
metrics such as coverage, simulation performance and
bug statistics

Metrics as part of the regression process


Metrics help ensure regressions are efficient & productive
Two types of regression metrics are often tracked:
Information on the regression run
Information on the simulation farm

Regression run information may include:


test names, frequency of tests, random seeds, configuration
choices and so on

By looking at the tests that provide the most coverage or


are most effective at uncovering bugs, test run order or
test frequency can be adjusted to increase verification
productivity

Metrics as part of the overall project


Useful non-simulation-based metrics related to bugs
Understanding bug status can provide insight into the simulation,
testing, coverage and overall progress
Knowing which simulation reported the bug
provides information about the effectiveness
of stimulus, checking, versions and abstraction
level used to find the bug
Knowing when the bug was closed can provide information about
the project progress and insight into when a particular test does not
need to be run in regressions

Additional project-wide metrics


Examples of other external measures of project-wide
progress
Code check-in rates (stability, maturity, churn)
Team status metrics (are teams getting everything they need)
Activity by geographic location
Simulation farm efficiency (bug rates per cycle)

Determining which metrics are useful is likely to depend on


many aspects of the project, the expected lifespan of IP
that is being developed and corporate culture

Summary
Metrics Considerations
Provide visibility into all aspects of build, stimulus,
check, coverage, and regressions.
Metrics must be actionable
Insight into the project generally comes from
correlation of measurements
Measurements from any part of a project may be
useful not just in the simulation

Metrics in SoC Verification


What Can Metrics Tell Us?

Andreas Meyer
Verification Architect
academy@mentor.com
www.verificationacademy.com