Pcie Basics Pcisig

PCI Express® Basics
Ravi Budruk
Senior Staff Engineer and Partner
MindShare, Inc.
Copyright © 2007, PCI-SIG, All Rights Reserved 1

PCI Express Introduction
PCI Express architecture is a high performance, IO
interconnect for peripherals in
computing/communication platforms
Evolved from PCI and PCI-X™ architectures
9 Yet PCI Express architecture is significantly different from its
predecessors PCI and PCI-X
PCI Express is a serial point-to-point interconnect
between two devices
Implements packet based protocol for information
transfer
Scalable performance based on number of signal
Lanes implemented on the PCI Express interconnect
PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 2

PCI Express Terminology
PCI Express Device A
Signal Link
Wire Lane
PCI Express Device B

PCI Express Throughput
Link Width x1 x2 x4 x8 x12 x16 x32
Aggregate 0.5 1 2 4 6 8 16
BW
(GBytes/s)
Assumes 2.5 GT/s signaling in each direction

80% BW utilized due to 8b/10b encoding overhead
Aggregate bandwidth implies simultaneous traffic in both directions
Peak bandwidth is higher than any bus available
PCIe 2.0 PHYs may optionally support 5 GT/s signaling, thus

doubling above bandwidth numbers

PCI Express Features
Point-to-point connection
Serial bus means fewer pins
Scaleable: x1, x2, x4, x8, x12, x16, x32
Dual Simplex connection
2.5VGT/s transfer/direction/s
Packet based transaction protocol
Packet
PCIe® PCIe
Device Link (x1, x2, x4, x8, x12, x16 or x32) Device
A B
Packet

Differential Signaling
Electrical characteristics of PCI Express signal
9 Differential signaling
– Transmitter Differential Peak voltage = 0.4 - 0.6 V
– Transmitter Common mode voltage = 0 - 3.6 V
D-
Vcm
V Diffp
D+
Two devices at opposite ends of a Link may support

different DC common mode voltages

Example PCI Express
Topology
CPU
Root Complex Memory
PCIe PCIe PCIe
PCIe PCIe PCIe

Switch
Endpoint Endpoint PCIe
Bridge To
PCIe PCIe PCI/PCI-X
PCIe Legacy
Endpoint Endpoint PCI/PCI-X
Legend
PCI Express Device Downstream Port
PCI Express Device Upstream Port

Example PCI Express
Topology – Root & Switch
CPU Bus
CPU
Root
RCRB
Bus 0
Root Complex Memory Virtual Virtual Virtual

PCI PCI PCI
Bridge Bridge Bridge
PCIe PCIe PCIe
PCIe PCIe PCIe

Switch PCI Express Links
Endpoint Endpoint PCIe
Bridge To
PCIe PCIe PCI/PCI-X
Switch
PCIe Legacy Virtual
PCI
Endpoint Endpoint PCI/PCI-X Bridge
Virtual Virtual Virtual

PCI PCI PCI
Legend Bridge Bridge Bridge
PCI Express Device Downstream Port

PCI Express Device Upstream Port

Example Low Cost
PCI Express System
Processor
FSB
PCI Express
GFX
GFX Root Complex
DDR
SDRAM
PCI Express Slots
Serial ATA PCI
HDD
USB 2.0 IO Controller Hub
(ICH) IEEE
LPC 1394
Slot
S
IO
COM1 GB
COM2 Add-In Add-In Add-In
Ethernet
PCI Express PCI Express

Link

Example PCI Express
Server System
Processor Processor
FSB
PCI Express
GFX
GFX Root Complex
DDR
Endpoint SDRAM
10Gb
InfiniBand
InfiniBand Switch Ethernet Switch Fiber
Switch
Endpoint Channel
“Out-of-Box”
Endpoint
RAID Disk array
10Gb PCI Express
Add-In Switch SCSI
Ethernet to-PCI
Endpoint Endpoint Endpoint
PCI
PCI Express
Link Gb Slots
Add-In IEEE
Ethernet S
IO 1394
Endpoint COM1
COM2
Transaction Types,
Address Spaces
Request are translated to one of four transaction types by
the Transaction Layer:
1. Memory Read or Memory Write. Used to transfer data from or to a
memory mapped location
– The protocol also supports a locked memory read transaction variant.
2. I/O Read or I/O Write. Used to transfer data from or to an I/O location
– These transactions are restricted to supporting legacy endpoint
devices.
3. Configuration Read or Configuration Write. Used to discover device
capabilities, program features, and check status in the 4KB PCI Express
configuration space.
4. Messages. Handled like posted writes. Used for event signaling and
general purpose messaging.

PCI Express TLP Types
Description Abbreviated
Name
Memory Read Request MRd
Memory Read Request – Locked Access MRdLk
Memory Write Request MWr

IO Read Request IORd
IO Write Request IOWr
Configuration Read Request Type 0 and Type 1 CfgRd0, CfgRd1
Configuration Write Request Type 0 and Type 1 CfgWr0, CfgWr1
Message Request without Data Payload Msg

Message Request with Data Payload MsgD
Completion without Data (used for IO, configuration write completions and read Cpl
completion with error completion status)
Completion with Data (used for memory, IO and configuration read completions) CplD
Completion for Locked Memory Read without Data (used for error status) CplLk
Completion for Locked Memory Read with Data CplDLk

Three Methods For
Packet Routing
Each request or completion header is tagged as to its type, and
each of the packet types is routed based on one of three
schemes:
9 Address Routing
9 ID Routing
9 Implicit Routing
Memory and IO requests use address routing.
Completions and Configuration cycles use ID routing.
Message requests have selectable routing based on a 3-bit
code in the message routing sub-field of the header type field.

Programmed I/O Transaction
Processor Processor
MRd FSB
Requester:
-Step 1: Root Complex (requester)
initiates Memory Read Request (MRd) Root Complex
-Step 4: Root Complex receives CplD DDR
SDRAM
MRd CplD
Switch A Switch C
MRd
CplD
Switch B Endpoint Endpoint Endpoint
MRd CplD
Completer:
Endpoint Endpoint -Step 2: Endpoint (completer)
receives MRd
-Step 3: Endpoint returns
Completion with data (CplD)
DMA Transaction
Processor Processor
FSB
Completer:
-Step 2: Root Complex (completer)
receives MRd Root Complex
-Step 3: Root Complex returns DDR
Completion with data (CplD) SDRAM
CplD MRd
Switch A Switch C
CplD
MRd
Switch B Endpoint Endpoint Endpoint
CplD MRd
Requester:
Endpoint Endpoint -Step 1: Endpoint (requester)
initiates Memory Read Request (MRd)
-Step 4: Endpoint receives CplD
Peer-to-Peer Transaction
Processor Processor
FSB
Root Complex
DDR
SDRAM
CplD MRd MRd CplD
Switch A Switch C
CplD
MRd MRd CplD
Switch B Endpoint Endpoint Endpoint Completer:

-Step 2: Endpoint (completer)
receives MRd
CplD MRd -Step 3: Endpoint returns
Completion with data (CplD)
Endpoint Endpoint Requester:
-Step 1: Endpoint (requester)
initiates Memory Read Request (MRd)
-Step 4: Endpoint receives CplD
PCI Express Device Layers
PCI Express Device A PCI Express Device B
Device Core Device Core
PCI Express Core PCI Express Core

Logic Interface Logic Interface
TX RX TX RX
Transaction Layer Transaction Layer
Data Link Layer Data Link Layer
Physical Layer Physical Layer
Link

TLP Origin and Destination

TX RX TX RX
TLP Transaction Layer Transaction Layer TLP

Transmitted Received
Link

TLP Structure
Information in core section of TLP comes
from Software Layer / Device Core
Bit transmit direction
Start Sequence Header Data Payload ECRC LCRC End

1B 2B 3-4 DW 0-1024 DW 1DW 1DW 1B
Created by Transaction Layer
Appended by Data Link Layer
Appended by Physical Layer

DLLP Origin and Destination

TX RX TX RX
DLLP Data Link Layer Data Link Layer DLLP

Link

DLLP Structure
Bit transmit direction
Start DLLP CRC End

1B 4B 2B 1B
Data Link Layer
ACK / NAK Packets

Appended by Physical Layer Flow Control Packets
Power Management Packets
Vendor Defined Packets

Ordered-Set Origin and Destination

TX RX TX RX
Ordered-Set Ordered-Set
Link

Ordered-Set Structure
COM Identifier Identifier Identifier
Training Sequence One (TS1)

9 16 character set: 1 COM, 15 TS1 data characters
Training Sequence Two (TS2)
9 16 character set: 1 COM, 15 TS2 data characters
SKIP
9 4 character set: 1 COM followed by 3 SKP identifiers
Electrical Idle (IDLE)
9 4 characters: 1 COM followed by 3 IDL identifiers
Fast Training Sequence (FTS)
9 4 characters: 1 COM followed by 3 FTS identifiers
Quality of Service
Processor Processor
PCI Express
GFX
GFX Root Complex
DDR
Endpoint SDRAM
10Gb
InfiniBand
InfiniBand Switch Ethernet Switch Fiber
Switch
Endpoint Channel
“Out-of-Box” Endpoint
RAID Disk array
10Gb PCI Express
Add-In Switch SCSI
Ethernet to-PCI
Endpoint Endpoint Endpoint
Slot PCI
Video Slots
SCSI S IEEE
PCI Express Camera IO 1394
Link Endpoint COM1
COM2

Traffic Classes and Virtual Channels
Quality of Service (QoS) policy through Virtual
Channel and Traffic Class tags
Transmitter Device A Receiver Device B

TC[2:0] Link
maps to
VC0
TC/VC Mapping
VC0 VC0
TC/VC Mapping
Arbitration
VC0
TC[7:0] Buffers Buffers TC[7:0]
VC1
TC[7:3]
maps to VC1
VC1
VC1 One physical Link,
multiple virtual paths

Port Arbitration
and VC Arbitration
Link
TC[2:0] to VC0 VC0

g
in 0
TC[7:3] to VC1 1 p
ap or
t
VC1 M sP Link
C s
C /V gre VC0
T rE Port VC0
fo
Arb
VC VC0
VC1
Arb 0
Port VC1
VC1
Link 0
rt g
Arb
o
P n
s pi
res ap
TC[2:0] to VC0 VC0 2
g M
r E /VC
o
f C
T
TC[7:3] to VC1 VC1

PCI Express Flow Control
Credit-based flow control is point-to-point
based, not end-to-end
Buffer space
available
TLP
VC Buffer
Transmitter Receiver
Flow Control DLLP (FCx)
Receiver sends Flow Control Packets (FCP) which are a type of DLLP (Data Link Layer Packet)
to provide the transmitter with credits so that it can transmit packets to the receiver

ACK/NAK Protocol Overview
Transmit Receiver
Device A Device B
From To
Tx Rx
TLP DLLP DLLP TLP
ACK / ACK /
Sequence TLP LCRC NAK NAK
Sequence TLP LCRC
Replay
Buffer De-mux De-mux
Error
Mux Mux Check
Tx Rx Tx Rx
DLLP
ACK /
NAK
Link
TLP
Sequence TLP LCRC

ACK/NAK Protocol: Point-to-Point
1a. Request 2a. Request

4b. ACK 3b. ACK
Requester Switch Completer
1b. ACK 2b. ACK

4a. Completion 3a. Completion
ACK returned for good reception of Request or Completion

NAK returned for error reception of Request or Completion

Interrupt Model: Three Methods
PCI Express supports three interrupt reporting
mechanisms:
1. Message Signaled Interrupts (MSI)
– Legacy endpoints are required to support MSI (or MSI-X) with 32- or
64-bit MSI capability register implementation
– Native PCI Express endpoints are required to support MSI with 64-bit MSI
capability register implementation
2. Message Signaled Interrupts - X (MSI-X)
– Legacy and native endpoints are required to support MSI-X (or MSI)
and implement the associated MSI-X capability register
3. INTx Emulation.
– Native and Legacy endpoints are required to support Legacy INTx
Emulation
– PCI Express defines in-band messages which emulate the four
physical interrupt signals (INTA-INTD) routed between PCI devices
and the system interrupt controller
– Forwarding support required by switches

Native and Legacy Interrupts
PCIe -
PCIe PCIe

PCI Express Error Handling
All PCI Express devices are required to support some
combination of:
9 Existing software written for generic PCI error handling, and which
takes advantage of the fact that PCI Express has mapped many of
its error conditions to existing PCI error handling mechanisms.
9 Additional PCI Express-specific reporting mechanisms
Errors are classified as correctable and uncorrectable.
Uncorrectable errors are further divided into:
9 Fatal uncorrectable errors
9 Non-fatal uncorrectable errors.

Correctable Errors
Errors classified as correctable, degrade system
performance, but recovery can occur with no loss of
information
9 Hardware is responsible for recovery from a correctable error and
no software intervention is required.
Even though hardware handles the correction, logging the
frequency of correctable errors may be useful if software is
monitoring link operations.
An example of a correctable error is the detection of a link
CRC (LCRC) error when a TLP is sent, resulting in a Data
Link Layer retry event.

Uncorrectable Errors
Errors classified as uncorrectable impair the functionality
of the interface and there is no specification mechanism
to correct these errors
The two subgroups are fatal and non-fatal
1. Fatal Uncorrectable Errors: Errors which render the link
unreliable
– First-level strategy for recovery may involve a link reset by the
system
– Handling of fatal errors is platform-specific
2. Non-Fatal Uncorrectable Errors: Uncorrectable errors
associated with a particular transaction, while the link itself is
reliable
– Software may limit recovery strategy to the device(s) involved
– Transactions between other devices are not affected

Baseline Error Reporting
Enabling/disabling error reporting
Providing error status
Providing error status for Link Training
Initiating Link Re-training
Registers provide control and status for

9 Correctable errors
9 Non-fatal uncorrectable errors
9 Fatal uncorrectable errors
9 Unsupported request errors

Advanced Error Reporting
Finer granularity in defining error type
Ability to define severity of uncorrectable errors
9 Either send ERR_FATAL or ERR_NONFATAL
message for a given error
Support for error logging of error type and TLP
header related to error
Ability to mask reporting of errors
Enable/disable root reporting of errors
Identify source of errors

PCI Express
Configuration Space

Summary of Changes for 2.0
Higher speed (5.0 GT/s), supported by:
9 Selectable de-emphasis levels
9 Selectable transmitter voltage range
Dynamic speed and link width changes
9 Power savings, higher bandwidth, reliability
Virtualization support
9 Access Control Services
Other New Features
9 Completion timeout control
9 Function Level Reset
9 Modified Compliance Pattern for testing

Thank you for attending the
PCI-SIG Developers Conference 2007.
For more information please go to

www.pcisig.com

PCI Express Basics
Ravi Budruk
Senior Staff Engineer and Partner
MindShare, Inc.
Copyright © 2007, PCI-SIG, All Rights Reserved 40

Pcie Basics Pcisig

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Pcie Basics Pcisig

Hochgeladen von

Copyright:

Verfügbare Formate

PCI Express® Basics

Copyright © 2007, PCI-SIG, All Rights Reserved 1

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 2

PCI Express Device A

PCI Express Device B

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 3

 Assumes 2.5 GT/s signaling in each direction

 PCIe 2.0 PHYs may optionally support 5 GT/s signaling, thus

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 4

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 5

 Two devices at opposite ends of a Link may support

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 6

Root Complex Memory

PCIe PCIe PCIe

PCIe PCIe PCIe

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 7

Root Complex Memory Virtual Virtual Virtual

PCIe PCIe PCIe

PCIe PCIe PCIe

Virtual Virtual Virtual

PCI Express Device Downstream Port

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 8

PCI Express PCI Express

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 9

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 11

Memory Read Request – Locked Access MRdLk

Memory Write Request MWr

Configuration Write Request Type 0 and Type 1 CfgWr0, CfgWr1

Message Request without Data Payload Msg

Completion for Locked Memory Read with Data CplDLk

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 12

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 13

Switch B Endpoint Endpoint Endpoint

Switch B Endpoint Endpoint Endpoint

MRd MRd CplD

Switch B Endpoint Endpoint Endpoint Completer:

Device Core Device Core

PCI Express Core PCI Express Core

Data Link Layer Data Link Layer

Physical Layer Physical Layer

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 17

Device Core Device Core

PCI Express Core PCI Express Core

TLP Transaction Layer Transaction Layer TLP

Physical Layer Physical Layer

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 18

Bit transmit direction

Start Sequence Header Data Payload ECRC LCRC End

Created by Transaction Layer

Appended by Data Link Layer

Appended by Physical Layer

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 19

Device Core Device Core

PCI Express Core PCI Express Core

DLLP Data Link Layer Data Link Layer DLLP

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 20

Start DLLP CRC End

Data Link Layer

 ACK / NAK Packets

PCI-SIG Developers Conference Copyright © 2007, PCI-SIG, All Rights Reserved 21

Device Core Device Core

PCI Express Core PCI Express Core

Data Link Layer Data Link Layer

Assumes 2.5 GT/s signaling in each direction

PCIe 2.0 PHYs may optionally support 5 GT/s signaling, thus

Two devices at opposite ends of a Link may support

ACK / NAK Packets

Training Sequence One (TS1)

Registers provide control and status for