Using FPGAs to Design Gigabit Serial Backplanes

April 17, 2002

Outline System Design Trends Serial Backplanes Architectures Building Serial Backplanes with FPGAs

A1 - 2

Key System Design Trends

Need for .
Easy Scalability of Performance Extremely High Availability Flexible Architecture Use of Standards
Rapid Time to Market Reduces Costs
A1 - 3

Merging Communications & Computers

Communications Systems have serial backplanes Computers moving to serial backplanes
Database servers Application servers 1U Web servers Ethernet Switches
Internet
A1 - 4

Bladed Server, PICMG 2.16

Serial Standards
Serial Standards
Fiber Channel 1 Gbps Ethernet 1 Gbps InfiniBand 2.5 Gbps

Parallel Standards Going Serial

PCI => 3GIO RapidIO => Serial RapidIO ATA => Serial ATA

Channel Bonded Serial Standards

XAUI 10 Gigabit Ethernet (4 x 3.125 Gbps)

3GIO 3GIO
A1 - 5

Serial Connectivity = Higher Bandwidth & Fewer Pins

1
50 Example - PCI: 32-bit x 33MHz = 1 Gbps , Shared among 5 clients 250 total pins

1
4

Virtex-II Pro System: 2.5 Gbps x 4 x 5 = 50 Gbps 80 total pins Each Client has 2.5 Gbps guaranteed to every other Client

50x higher bandwidth with less than 1/3 of the pins

RJG
A1 - 6

Serial Link Rates

Link Rates (bits per sec.)
40G 10G 3.125G 1.25G 622M 155M OC-768 OC-192 10GE FC-10x SxI-5 3GIO Infiniband FC-2x Rapid IO

OC-48 FC OC-12 OC-3 1998 2000 GigE

Mainstream Deployment

2002

2004

2006

A1 - 7

Serial Topologies
Switched Full Mesh

PICMG 2.16

PICMG 2.2 PICMG 3.x

A1 - 8

Building Serial Backplanes with FPGAs

Switched

Virtex-II

Gigabit Ethernet Phy x2 Gigabit Ethernet MAC x2

PICMG 2.16

A1 - 9

Virtex-II Pro
Platform FPGA

A1 - 10

Virtex-II Pro Rocket I/O Technology

Physical Coding Sublayer Physical Media Attachment

(PCS) 32 / 16 / 8 bits
TXDATA C R C 8B / 10B Encode F I F O

(PMA)
Serializer Transmit Buffer

TX+ TX-

TX Clock Generator

40 156.3 MHz REFCLK

20X Multiplier
Receiver

32 / 16 / 8 bits
RXDATA

CRC Elastic Buffer

Channel Bonding and Clock Correction

RX Clock Recovery
Deserializer Comma Det.

8B / 10B Decode

Receive Buffer

Loop-back

Transmitter

RX+ RX-

Up to sixteen multi-gigabit serial transceivers Support 622 Mbps to 3.125 Gbps full duplex operation Flexible PCS/PMA feature set
A1 - 11

Xilinx Rocket I/O Serial Backplane Interface Technology

Rocket I/O Serial Backplane Interface (RSBI)
Technology
Simple solution for serial backplane applications Consists of two parts
The RSBI/ Aurora link protocol An RSBI Reference Design (IP core) implementing
that protocol

A1 - 12

RSBI/ Aurora Protocol

An Efficient, Lightweight Protocol to Move
Data Point to Point across Serial Link Used to transport higher-level protocols Standard protocols like Ethernet Proprietary, customer-defined, protocols

A1 - 13

RSBI Core
Implements RSBI/ Aurora Protocol Provides a simple interface for transferring packets across
A single gigabit serial link Up to 16 bonded links

Minimal Use of Resources (~350 slices) Frequency and Technology Independent Specifications and Source Code available from Xilinx
A1 - 14

RSBI/ Aurora Features

Uses 8B/10B and includes CRC protection Transmitter wraps user data in Rocket I/O Aurora protocol
Handles packet formation Sends error notification to remote core

Receiver recovers frames and realigns data

Strips packet/frame wrappers Aligns data on double word boundary Monitors error conditions

A1 - 15

RSBI User Interface

RSBI 32 SIMPLE 32 COMPLEX CONTROL 32 Rocket I/O Block

USER

CONTROL 32

Very High Speed Data

User interface created so no knowledge of Rocket I/O block is necessary

A1 - 16

RSBI Channel Bonding

All lanes must have passed comma detect before channel bonding starts Uses same technique as 10 Gigabit Ethernet and Serial RapidIO
Sends semi-random IDLE pattern
Virtual data channel

Channel Bonding Logic

Synchronization of individual channels into a single large data channel

A1 - 17

RSBI/ Aurora Implementation Parameters

Allows maximum frame size of 8K Unlimited bytes 32-bit internal interface Other frame sizes and internal interface widths planned in future releases

A1 - 18

Error Detection in Hardware

Errors can be either in Rocket I/O block or link(s) Five sources of link failure detected:
Link physical error threshold overflow Channel misalignment FIFO overflow
Receiver or transmitter

Transmitter sending an illegal K character Upon detection of link failure, core resets itself and forces link partner to reset
A1 - 19

RSBI Transmission
All data frames have a SOP/SOF and EOP/EOF to
indicate the start and end of packet

User must send at least 2 DWORDs (8 bytes) Core will add pad data to allow CRC functionality User can not transmit data in increments less than input
width Data can now be transmitted in any increment

A1 - 20

RSBI Reception
Strips off SOP/SOF* and EOP/EOF**
from frame

Strips off pad data if necessary Signals any errors detected

Checks CRC for correctness
*SOP/F = Starting of Packet/Frame **EOP/F = End of Packet/Frame
A1 - 21

Serial Backplane: Full Mesh

2VP50

Design Approach Supporting 16 slot, 3.125 Gbps per link Full Mesh Serial Backplane RSBI layer Rocket I/O Transceivers

X 15

A1 - 22

Implementing RSBI in Virtex-II Pro Platform FPGA

Rocket I/O Transceiver Rocket I/O Interface User Interface

RSBI Link
FPGA Editor View of 2VP7

Auto Negotiation & Synchronization

Framer Framer User App

Implementation in Virtex-II Pro FPGA Features

Up to 40 Gbps 32-bit user interface Low overhead

Auto Negotiation & Synchronization

Framer Framer

RSBI Core

Utilization
350 Slices

A1 - 23

Serial Backplane:Private Connection

2VP50 2VP2

X 15

X4

Parallel Connection on line card

4 Transceivers Channel Bonded for Private board to board connection

A1 - 24

Sample Serial Backplane Approaches

Switched
2VP20

XAUI Core Rocket I/O Transceivers 4

XAUI Switch

4
XAUI Switch

Backplane

A1 - 25

Implementing XAUI in Virtex-II Pro FPGA

XAUI Core
TXD, TXC

XAUI Floorplan 2VP7

Transmitter
State Machine

XAUI_TX_L0 XAUI_RX_L0 XAUI_TX_L1 XAUI_RX_L1

Rocket I/O Transceivers

Receiver

Deskew

RXD, RXC

Sync Sync Sync Sync

XAUI_TX_L2

Implementation in V-II Pro

Uses 4 Rocket I/O Transceivers
Channel bonding 8B/10B encoding

XAUI_RX_L2

XAUI_TX_L3 XAUI_RX_L3

Utilization
800 Slices

Management Ports

A1 - 26

Signal Integrity Design Resources

http://www.xilinx.com/signalintegrity

A1 - 27

Power Filtering Network

A1 - 28

Power Filtering Network Components

Capacitors
C = 0.22uF Package: 0603 SMT Dielectric: X7R Tolerance: 10% WVDC: 10V Spacing: Within 1cm of pin

Ferrite Beads
Murata BLM11A102S

Voltage Regulator
Linear Technology LT1963

All parts specified in PCB Design Requirements Voltage Regulator must use circuit specified by manufacturer

A1 - 29

VTTX, VTRX
Top of board

Ferrite Bead

Capacitors

A1 - 30

AVCCAUXTX, AVCCAUXRX
Bottom of board Ferrite Bead Capacitor

FPGA Bypass Caps

A1 - 31

Reference Clock
Must use EPSON EG-2121CA
2.5V LVPECL output oscillator Use according to manufacture specifications

Resistor network for clock input:

R1 R2 R3 R4 510 ohm 130 ohm 25 ohm 100 ohm

A1 - 32

Implementation Resources
Physical (PCB) Design Information
Signal Integrity Central on Xilinx Web Site (http://www.xilinx.com/signalintegrity) Rocket I/O User Guide Spice Suite

RSBI Design Information (http://www.xilinx.com/rsbi)

Specification, Implementation notes Source code (Verilog) Test bench

XAUI information (http://www.xilinx.com/connectivity)

In Networking/Datapath Products (10 Gigabit Ethernet)

A1 - 33

Thank You