QEMU based Co-simulation platform -

Benefits
By Santanu Bhattacharyya, Rohit Chaturvedi, Bhupendra singh, Nitin Goel (Solarflare
Communications)

1 Introduction

Semiconductor industry is growing with tremendous pace. Silicon size,

features and software complexity have increased to a never before level.
Time to market has been reduced and the demand for quality of product
has increased. Once the product definition is available the development cycle starts.
Development cycle has many phases like architecture, design, verification, software
development and testing, emulation and silicon validation.

To meet the time-to-market target, there is always a need to bring parallelism to the product
development cycle. Though design and verification are done in parallel, the software
development generally starts a bit late and continues till product delivered to the end customer.
As software is not tested fully before the silicon fabrication, this leads to re-spin of silicon
sometimes and also leads to delay of the product delivery.

With the advent of constraint based random simulation, verification has become more robust
from its earlier versions. While these random test-benches are helpful to cover corner cases at
lower layers of abstraction, it is not enough to find system level behaviour of the design when it
is integrated with software. For this it is necessary to run simulation of hardware logic along with
software. This is where hardware/software co-verification using SystemVerilog Direct
Programming Interface (DPI) comes into picture. Mostly DPI used to provide access of C/C++
libraries to SystemVerilog. Co-Verification setups use DPI for interfacing hardware with
software. In most cases, SystemVerilog/UVM is used to generate stimulus which is then
transferred to either a (SystemC based) virtual platform or an emulation platform. In this
scenario, the DPI is used to drive software transaction to the DUT. QEMU(Quick Emulator)
based virtual platform has been integrated with SystemVerilog based RTL simulation
environment. Transactions are generated by actual software (Application) running on QEMU.
The SV environment captures these transactions through DPI and drives them on the DUT
interfaces. This paper proposes a flow which illustrates how to develop and test software early in
the development phase along with design-verification cycle. Also this paper highlights the
benefits of the flow which will not only reduce the development cycle but also improve the
quality of product.

2 Overview of DUT (Design under Test)

Traditionally, network traffic is processed on processor where there is a NIC (Network Interface
Card) interfacing the network i.e. wire side through Ethernet interface and software taking care
of the packet processing. There are numerous applications where Processor based packet
processing might be inadequate to meet the processing latency requirement. Applications such as
High frequency trading and online advertisement demand extremely low processing latencies. To
meet this type of latency requirement it is better to process the packet on a dedicated hardware
along with the NIC and communicate to the software processor farm. The dedicated hardware
i.e. the DUT is shown in Figure 1 below as co-processor to NIC having Ethernet interface on
both the NIC and the wire side. The NIC interfaces with the host over PCIe bus and on the
network side it has Ethernet interface. The DUT in addition to the Ethernet interface also has
host interface for configuration and device status monitoring purpose.

Figure 1: DUT Block Diagram

3 System Verification Environment

In the Verification Environment shown in figure 2 below has Packet generator which can be a
system Verilog packet generator or user world application called PackETH. PackETH provides
GUI interface to customise the packets by controlling addresses, payloads, length of packet,
packet type etc. Apart from these it can also provide corrupted packet to check error scenarios.

Tap interfaces (virtual interfaces created by Linux command tunctl) are special software entities
which instruct the Linux machine to forward Ethernet frames. In other words, the virtual
machines connected to tap interfaces will be able to transmit and receive raw Ethernet frames.
And due to this virtual machines can continue to emulate physical machines from a networking
perspective.

C-Layer has tap sniffers which acts as interface between DEQUE & tap interface. DEQUE is a
temporary storage queue in which it takes data from both direction transmit as well as receive. C-
Layer communicates to QEMU through TCP sockets.

Streaming Data Adapter (SD) converts Ethernet frames to be compatible with DUT proprietary
interface and passes it to DPI interface.
UVM Environment instantiates the DUT, Monitors, Scoreboards, Models and Drivers. In early
stage of design cycle when the DUT is not available, a DUT model in place of actual DUT (in
C++/UVM) can be integrated to start the early development of Software. The DUT model can be
developed in C++ or UVM.

QEMU is a hosted virtual machine. It emulates CPU and provides a set of device models,
enabling it to run a variety of unmodified guest operating systems. QEMU has Hardware
Abstraction Layer (HAL) layer and it runs the Host side application. HAL is a hardware
abstraction layer which is provides API to configure design (DUT).

Socket Adapter is used for inter-process communication and it uses the client server model. Here
QEMU is Client and C-Layer is server. The client connects to the server for exchange of
information. QEMU interacts with C-Layer using socket adapters for configuring the DUT,
transmit packet to the DUT and receive packet from the DUT.

Control and Data Flow:

Control path transaction is generated by the application software running on QEMU and these
transactions through the DPI calls will be captured by the UVM environment and there it will be
driven on DUT on various interfaces.

PackETH generates Ethernet traffic which is pumped into the tap interfaces. This tap interfaces
captures the packet in D-queue inside C-layer and then the packet is passed onto the UVM
Environment through DPI calls. At NIC side processed packet (output packet from DUT) is
passed to C-layer which connects to QEMU through sockets.

Figure 2: System Verification environment with DUT

3.1 Software development Environment

Figure 3: Software development environment with C++ model

At the start of project when actual hardware is still under development a software model of the
DUT is developed and integrated. The software model is either untimed (written in C/C++) or
(when performance benchmarking is required) a loosely timed model written in SystemC. Most
hardware designs constitute multiple interfaces to the external world so C++ model has to
support multiple ports.

3.2 Software Model Architecture

Linux allows two types of shared-memory threading: Pre-emptive threading (posix threads) and
Cooperative threading. In Cooperative threading single thread is executed at a time so a single
processor is utilized. Pre-emptive threading multiple threads can run at one time so Operating
system is required to control the operation. Posix threads are mostly used in embedded software
especially software working with multiple interfaces. . Since there are multiple interfaces
which are active simultaneously, a separate posix thread is dedicated to each interface. Since
multiple threads run simultaneously and use shared memory, software races are possible. Special
care is required to avoid race conditions.

Concurrency issues faced with multithread applications are: Thread Races and Thread
Starvation. Thread race occurs if two threads simultaneously try to access shared data. Mutex
module is used from C++ boost library to avoid race conditions. To make threads efficient, a
separate Mutex lock is created for each shared data queue.

3.3 SV Simulator interfacing issue with Software Multithread

When Actual design comes DUT (with UVM Environment) is integrated in the environment as
shown in figure 2. Since these are two different system which has inherent challenges to
integrate. System Verilog(SV) Simulators predominantly execute on single thread. So the
question arises as to how to integrate SV Simulator, single threaded system, with software multi-
threaded system. Solution is to design a software module which takes care of multiple threads
and socket handling. It is named as Multithread Handler as shown in figure 4. This
multithreaded handler is present in socket adapter block. Simulator through DPI calls, attaches
with the main thread of the multi-thread handler, which in turn spawns all other independent
threads required for handling socket and virtual interface data. Main thread waits for notification
and then locks each queue one by one looking for transaction. Mutex locking has been used for
it. Mutex, helps protect shared data and avoid thread races. To make threads efficient, a separate
Mutex lock is created for each shared data queue. Since all the forks are executed on the same
processor in a determined order, there is no scope of software race conditions.

Figure 4: Transaction Flow

4. Benefits of Flow

4.1 Development Cycle

1. Development cycle reduction: Development time will be reduced as software

development and system test can be addressed early in development phase.
2. DUT model: Since it is plug-n-play environment, DUT can be replaced by the DUT
model. Earlier verification and software team used to develop different model, but now
only one model can be developed which can be used by both the team. Same model can
be used for Software unit test, hardware block level test and for system test.
3. Single Environment for Unit to system level test: The environment will be seamlessly
used for Software Unit testing and System level testing.
4. Legacy software compatibility check: Most of the time customers who are using enhance
product wants the legacy software to be back-compatible. This flow will enable to test
customer legacy software and any issue can be fixed upfront.
 5. Post Silicon support: Issues reported by customer while running on field can easily be
replicated and checked without changing customer software. Same software which
customer is running can be easily ported and verified on this platform.

4.2 Quality

1. Customer use cases: System level scenarios supporting customer use cases can be run to
improve the quality of design.
2. . Robust Generator: Many a times design breaks when tested against actual data
generators. This flow will help to plug traffic generator utility like PackETH which
generates bus traffic very close to the traffic generator used in the field.

5. Conclusion

SW driven mechanism provides a convenient way to drive transactions at DUT through higher
levels of abstraction. By enabling the hardware test along with Software, it not only improves the
product life cycle but also the quality of deliverables. It helps to avoid finding design or
architecture issues at late stage. It also provides the platform to test the legacy software and find
issues upfront.

6 ACRONYMS

QEMU Quick Emulator

UVM Universal Verification Methodology
SV System Verilog
DUT Design under Test
GUI Graphic User Interface
NIC Network Interface Card
API Application Programming Interface

7REFERENCES

A. F. et al, Hardware/software co-verification using the SystemVerilog dpi. DVCon,

2013
S. Sutherland, Integrating SystemC models with Verilog using the SystemVerilog direct
programming interface. SNUG, 2004

If you wish to download a copy of this white paper, click here