Harraser and Packet analysis for

Gigabit Ethernet in Post Silicon Server

Validation
October 30, 2014

Interrupts

1.1

Introduction

An Interrupt is a hardware signal from a device to a CPU,informing the CPU that

the needs attention and signalling that the CPU should stop current processing and
responding to the device.
If the CPU is performing a task that has lower priority then the priority of the interrupt,the CPU suspends its current thread.The CPU then invokes the interrupt handler
for the device that sent the interrupt signal.The interrupt handler services the device and
when the interrupt handler returns,the CPU resumes the processing it was doing,before
the interrupt occured.
First we will see how devices do interrupts in a legacy PC.

Figure 1: Older Way of Interrupts

Legacy interrupts need Multi-step communication to execute an interrupt. The
required steps need are as follows.
A device signals that it needs CPU service
The Interrupt Controller signals the CPU
The CPU responds with INTA
INTA puts ID-number on system bus
CPU uses ID-number to lookup IVT entry
Interrupt handler executes Interrupt service routine and returns where it was interrupted.

1.2

Improvement to be done for faster Interrupt handling

Faster response to interrupts is possible if the old multi-step communication scheme

can be replaced by a single-step protocol. Less expensive PCs can be manufactured if
their total number of signal pins and the physical interconnections can be reduced. More
devices can have their own private interrupt(s) if signal lines arent required So this brought
the need for development of a new system for handling interrupts in a fast and efficient
way.So MSI were developed for this.Message Signaling allows all the needed information
to arrive in a single package, and go directly from a device to the CPU.
2

Figure 2: New Way of handling Interrupts

1.3

Introduction to MSI and Its Capabilities

Message Signaled Interrupts (MSI) are an alternative in-band method of signaling an

interrupt, using special in-band messages to replace traditional out-of-band assertion of
dedicated interrupt lines. While more complex to implement in a device, message signaled
interrupts have some significant advantages over pin-based out-of-band interrupt signaling
Message signaled interrupts are supported in PCI bus since its version 2.2, and in
later available PCI Express bus. Some non-PCI architectures also use message signaled
interrupts[1]
Traditionally, a device has an interrupt line (pin) which it asserts when it wants to
signal an interrupt to the host processing environment. This traditional form of interrupt
signaling is an out-of-band form of control signaling since it uses a dedicated path to
send such control information, separately from the main data path. Message signaled
interrupts are replacing those dedicated interrupt lines with in-band signaling, where
special messages indicating interrupts are exchanged through the main data path.
As an example, PCI Express does not have separate interrupt pins at all, and
it uses special in-band messages to allow it to emulate an interrupt pin assertion or
deassertion. Message signaled interrupts allow the device to write a small amount of data
to a special memory-mapped I/O address,the chipset then delivers the corresponding
interrupt to a processor.
A common misconception with Message Signaled Interrupts is that they allow
the device to send data to a processor as part of the interrupt. The data that is sent as
part of the write is used by the chipset to determine which interrupt to trigger on which
processor; it is not available for the device to communicate additional information to the
interrupt handler.

1.4

MSI Types

PCI defines two optional extensions to support Message Signaled Interrupts.

They are
MSI
MSI-X

1.4.1

MSI

MSI permits a device to allocate 1, 2, 4, 8, 16 or 32 interrupts. The device is

programmed with an address to write to (generally a control register in an interrupt
controller), and a 16-bit data word to identify it. The interrupt number is added to the
data word to identify the interrupt. Some platforms such as Windows do not use all 32
interrupts but only use up to 16 interrupts[2]
Figure.3 shows how ethernet traffic is distributed across single CPU in a singlecoresystem.

Figure 3: Network Data Flow In a Previous Generation Platform

1.4.2

MSI-X

MSI-X is an extension to MSI to enable support for more vectors and other advantages.MSIX permits a device to allocate up to 2048 interrupts. The single address used by original
MSI was found to be restrictive for some architectures. In particular, it made it difficult
to target individual interrupts to different processors, which is helpful in some high-speed
networking applications. MSI-X allows a larger number of interrupts and gives each one
a separate target address and data word. Devices with MSI-X do not necessarily support
2048 interrupts but at least 64 which is double the maximum MSI interrupts. Optional
features in MSI (64-bit addressing and interrupt masking) are also mandatory with MSIX.
4

The ability to communicate efficiently between queues and particular processor

cores is handled by MSI-X. MSI-X is the next generation of MSI, which passes interrupts
to a single processor core. Conversely, MSI-X provides multiple interrupt vectors, which
allow multiple interrupts to be handled simultaneously and loadbalanced across multiple
cores. This improvement helps improve CPU utilization and lower latency[3]
With an interrupt vector for each queue, the controller can handle multiple interrupts simultaneously, preventing the bottlenecks associated with guiding all interrupts
through a single vector.
Figure.4 shows how ethernet traffic is distributed across CPU cores in a multi-core
system.

Figure 4: Network Dataflow using MSI-X in a Multi-Core System

References
[1] http://www.intel.in/content/dam/www/public/us/en/documents/whitepapers/msg-signaled-interrupts-paper.pdf.
[2] B. H. Leitao, Tuning 10gb network cards on linux, in Proceedings of the 2009 Linux
Symposium, 2009.
[3] http://www.intel.in/content/dam/doc/white-paper/improving-networkperformance-in-multi-core-systems-paper.pdf.