Packet-Buffering Inside Routers

Strategies
of packet buffering

inside Routers

Rafal Jan Szarecki #JNCIE136
Solu@on Architect, Juniper Networks
Why should I care ?
•  Under load, Your router’s discard behavior
according to queuing strategy selected by
vendor.
–  Could be quite unintui@ve !
–  BePer to know, how to live/deal with this ar@ficial
“intelligence”. And turn it for your benefit.
–  Do not troubleshoot if there is nothing
unexpected/misbehaving.
How it manifest
Traffic loss
•  Something is going on
–  SLA monitoring system rise
alarm
–  Customer calls and
complains
•  Which node in network
cause it?
–  You may need go node-‐by
node.
–  Other expert/anali@c
systems my help.
–  Out of scope
Too high jiPer/latency •  When guilty node is nail
down …
Unintui@ve behavior
Port u'liza'on ~1%
100GE
& traffic loss !!!
10GE
port
port
1 Gbps <1Gbps
“wirespeed”
router
Other Other
traffic traffic
Port u'liza'on 100%, port

FLOW A buffer size is Xms
FLOW A’
port But:
+
“wirespeed” •  Unfair Latency:
FLOW B’
router Flow B << Flow A
FLOW B
•  Latency is > Xms
Other Other
traffic traffic
Defect (bug) or expected behavior ?

Queuing != QoS
•  Queuing goal – avoid traffic/
packet drops during temporal
conges@on tx-rate reduction, class blind -
red and blue
Queueing impacted proportional
vs.
QoS
tx-rate reduction, class aware -
blue impacted more
•  QoS goal -‐ provide differen@al

treatment and separa@on
among traffic of different
classes.
–  Avoid traffic/packet drops during
temporal conges@on in some classes at
expenses of losses in other.
–  Re-‐order packets in a way to deliver data
of some classes as fast as possible.
This talk is not on QoS.

We will look at best-‐effort only
Router anatomy
Router is the embedded network

MUX SWITCH FABRIC DEMUX

Ingress Egress
PFE PFE
Ingress Egress
PFE PFE
PFE – a CPU, NPU or ASIC that process packet

The MUX
•  Mul@ple low-‐speed
In (Rx) port and
N x Rx
1 x Tx
•  Single high-‐speed
Out (Tx) interfaces
•  Many to One
•  No conges@on risk –
N x Rx rate <= 1 x Tx rate no need for buffer
The de-‐mux
•  Simple model
•  High-‐speed In (Rx) port
and
•  Mul@ple lower-‐speed
N x Tx
Out (Tx) interfaces
1 x Rx
•  One to many
•  1 Rx to 1 Tx @ same

Rx rate == N x Tx rate @me è Conges@on.
Queuing architectures for de-‐mux
•  Simple Output
Queuing – OQ
IF0 Tx
•  Usually implemented
Pn RX
IF1 Tx in single (shared)
memory
IFn Tx
The (asynchronous) switch fabric
•  N x In (Rx) and N x Out (Tx)
port of switch of same
speed
•  Any to Any
•  Each ingress port is
independent
N x Rx
N x Tx
–  Traffic/datagram may
appear at any @me
–  Not aware about egress
port state
•  N Rx to 1 Tx @ same @me

Rx rate == Tx rate è Conges@on.
Queuing architectures for switch fabric
•  Output Queuing – OQ – not used due to
technological limita@ons.
•  Input Queuing – IQ
•  Virtual Output Queuing -‐ VOQ

IQ
•  <100% efficiency
Port P1 Rx Port P1 Tx
•  Queue fan-‐out need
to be over 2 x
P1 OQ
desired port traffic to

P2 OQ
get 99%+ efficiency

Port Pn Rx Port Pn Tx
Pn OQ
IQ Flow-‐control
GR
•  Asynchronous – each
egress is independent
•  Ingress PFE sends Request
RQ
RQ –  Each has data size

Port P1 Rx
P1 OQ
Port P1 Tx
–  Only for packet at head of
queue
•  egress PFE answer w/ Grant
P2 OQ
– when egress Fabric port is

free
Port Pn Rx Port Pn Tx •  Egress schedules grants
–  Prevent starva@on
Pn OQ
–  E.g. RoundRobin or fair-‐

share
HoL blocking
Can’t be send, even P2 Tx is free
VOQ
•  Variant Input Queuing –
RQ
RQ
“dedicated Input queue
RQ
RQ
for each output port”
•  No need for over-‐speed
RQ
P1 VOQ
P2 VOQ RQ Port P1 Tx
Port P1
Rx
•  Flow-‐Control and
Pn VOQ
P1 VOQ
Port P2 Tx
scheduling
Port P2 P2 VOQ
Rx
Pn VOQ
–  Extension to IQ

P1 VOQ
P2 VOQ
–  Ingress PFE can send

Pn VOQ
P1 VOQ
Port Pn Tx
requests to mul@ple

Port Pn P2 VOQ
Rx
Pn VOQ
egresses simultaneously

VOQ
GR
GR
Port P1
Rx
P1 VOQ
P2 VOQ GR Port P1 Tx •  No need for over-‐speed
Pn VOQ
P1 VOQ
Port P2 Tx
scheduling
Port P2 P2 VOQ
Rx
Pn VOQ
P1 VOQ
P2 VOQ –  Extension to IQ
Pn VOQ
P1 VOQ
grants to mul@ple

Port Pn P2 VOQ
Port Pn Tx
Rx
Pn VOQ

VOQ
Port P1
Rx
P1 VOQ
P2 VOQ Port P1 Tx •  No need for over-‐speed
Pn VOQ
P1 VOQ
Port P2 Tx
scheduling
Port P2 P2 VOQ
Rx
Pn VOQ
P1 VOQ
P2 VOQ –  Extension to IQ
Pn VOQ
P1 VOQ
grants to mul@ple

Port Pn P2 VOQ
Port Pn Tx
Rx
Pn VOQ
•  No HoL blocking
The router
Line Card Line Card
ingress part egress part
Switch Fabric
Overload possible.
Multiple Rx ports
Buffer Buffer
memory Fabric ports memory sends to single Tx
port
Ingress PFE egress PFE
ASIC/NPU Switch Fabric P1 Tx ASIC/NPU
P1 Rx
switch
P2 Rx P2 Tx
Overload possible.
Router ingress Hi-speed fabric port
Interfaces à low-speed
Pn Tx
Interface
Pn Rx
•  Mul@stage •  hiSpeed (fab) to low speed port

–  Ingress mux to fabric •  Conges@on points
•  No conges@on –  Fabric-‐out (manyà one)
–  Fabric switch –  Egress mux (fast à slow)
–  Egress demux to ports –  Need queuing
Two approaches
Buffer twice – CIOQ systems Buffer Once -‐ VOQ systems
Combined Input Output Queuing Virtual Output Queuing
•  Buffer before fabric; de-‐queue when •  Buffer before fabric;
fabric egress port is available (empty) – De-‐queue when all way down to egress
fabric VOQ (or IQ) interface is available (empty) – end-‐to-‐
•  Buffer before egress interface; de-‐ end system VOQ
queue when interface is available
(empty) – OQ
•  Simpler to Implement •  Requires a lot of queues -‐ complex queue
•  Higher scalability [O(n)] management @ scale [O(n2)]
•  Requires more memory •  Requires less memory
•  Space (size) •  Space (size)
•  2 x bandwidth •  1 x bandwidth
•  Bigger system residency @me and JiPer •  Lower residency @me (latency inside router)
•  Lower power requirements
Buffer twice – latency
RQ
RQ
RQ
RQ
IF0 Tx
RQ
P1 VOQ IF1 Tx
P2 VOQ RQ Port P1 Tx Pn RX
Port P1
Rx
Pn VOQ
IFn Tx
P1 VOQ
Port P2 Tx
Port P2 P2 VOQ
Rx
Pn VOQ
•  Queues == Buffer ==
P1 VOQ
P2 VOQ
accommodate burst and delay it
Pn VOQ •  Burst absorp@on capability
Port Pn
P1 VOQ
Port Pn Tx depends on type of burst -‐ at
which point conges@on appears
P2 VOQ
Rx
Pn VOQ
–  Max: Σ (VOQ size, OQ size)

–  VOQ_size or OQ _size if
conges@on in one point only.
•  Max latency: Σ (VOQ size, OQ size)
Poten@al conges@on point •  Wait, there will be example
Buffer twice – bandwidth
Fabric Scheduler and flow-‐control
•  Which ingress PFE get
GRANT next. (e.g. fair-‐
P1 VOQ IF0 OQ
P2 VOQ
IF1 OQ
Pn VOQ IFn OQ share) from given egress

PFE
IF0
IF1 •  Monitor Fabric Egress and
P1 VOQ stop giving GR from queue
on Fabric ingress that
P2 VOQ

suppose to egress fabric
Pn VOQ
IFn
ASIC/NPU ASIC/NPU
via congested port
•  Packet received from

Ingress PFE
ASIC/NPU
Fabric are
–  stored in egress port
Buffer Buffer
memory Fabric ports memory
output queue.
P1 Rx
switch
P2 Rx P2 Tx
Router ingress
Interfaces
–  Or dropped if queue is
Fabric scheduler
Pn Rx
Pn Tx
full.
•  Fabric queuing and flow-control independent form egress port queuing and scheduling.
•  Router behavior – residency time (latency), jitter, drop rate depends on both.
End-‐to-‐End VOQ system
Buffer Buffer
memory Fabric ports memory

P1 Rx
switch
P2 Rx P2 Tx
Router ingress
Interfaces
Pn Tx
Pn Rx
•  For queuing purpose, switch fabric and all de-‐mux – seen as single switch
–  N inputs (Rx) à M output (Tx)
–  N x Rx speed == M x Tx speed
–  M >> N;
Buffer once -‐ latency
RQ/GR
RQ
RQ RQ/GR
RQ
RQ
RQ/GR RQ/GR
IF0 Tx
Port P1
P1 IF0 VOQ
P1 IFn VOQ Port P1 Tx IF1 Tx IF0 Tx
Rx
Pn IFn VOQ
Port P1 Tx
IFn Tx IF1 Tx
P1 IF0 VOQ RQ/GR
Port P2 P1 IFn VOQ Port P2 Tx
Rx
Pn IFn VOQ RQ
IFn Tx
P1 IF0 VOQ
RQ/GR
P1 IFn VOQ
Pn IFn VOQ
Port P2 Tx
Port Pn
P1 IF0 VOQ
P1 IFn VOQ
Port Pn Tx •  Queues == Buffer == accommodate
burst and delay it
Rx
Pn IFn VOQ
RQ
•  Burst absorp@on capability DO
NOT depends on type of burst
(regardless at which point
conges@on appears)
–  Max: VOQ size
–  Min: VOQ size
Port Pn Tx
•  Max latency: Σ (VOQ size)
Poten@al conges@on point
•  Wait, there will be example
Buffer once – bandwidth
Fabric Scheduler and flow-‐control
P1 IF0 VOQ
P2 IF0 VOQ •  Fabric Queuing -‐ VOQ (per
P1 IF1 VOQ
Pn IF0 VOQ
P2 IF1 VOQ
egress interface)
P1 IFn VOQ
Pn IF1 VOQ
P2 IFn VOQ •  Fabric scheduler and Flow-‐
Pn IFn VOQ
IF0 Control
IF1
P1 IF0 VOQ –  Which ingress [PFE, VOQ]
P2 IF0 VOQ
get GRANT next. (e.g. fair-‐
Ingress PFEP1 IF1 VOQ
Pn IF0 VOQ
P2 IF1 VOQ egress PFE share) from given egress
ASIC/NPU IFn
P1 IFn VOQ
Pn IF1 VOQ
P2 IFn VOQ
ASIC/NPU PFE
Pn IFn VOQ –  Monitor egress interface
and stop giving GR from
queue on Fabric ingress if
Ingress PFE egress interface is not
ASIC/NPU Buffer
memory Fabric ports
Buffer
memory
free
•  Packet received from

P1 Rx
switch
Fabric is immediately send

P2 Rx P2 Tx
Router ingress
out by egress interface

Interfaces
Pn Tx
Fabric scheduler
Pn Rx
•  Fabric queuing and flow-control depends on egress interface only.

System characteris@c
vs. Queuing architecture
CIOQ CIOQ (w/ fabric E2E VOQ
VOQ)
Low residency @me ✖ ✖ ✔
High load ✖ ✔ ✔
Low power footprint ✖ ✖ ✔
High number of interfaces ✔ ✔ ✖
(each with independent
queuing. E.g. BNG, BE)
mul@-‐chassis systems ✔ ✔ ✔
Examples* C7500, Juniper MX, Juniper PTX
Early C7600 Cisco ASR9k, Cisco NCS600
CRS-‐X
ALU 7750/7950*
* Please contact me if you want to update, correct add more – rafal@juniper.net
UNINTUITIVE BEHAVIOR -‐
BANDWIDTH
Intui@ve drop behavior
100GE
F1 150G F1 – F5 600/700 è 86% loss expected
F2 50G F6
100GE
700G
F3 200G
F4 100G Intui've (loss)? Observed 2 (router X) Observed 2 (router Y)
Gbps Loss % Gbps Loss % Gbps Loss %

F5 200G
F1 21 86%
F6 1G
F2 7 86%
F3 29 86%
•  All traffic is BE
F4 14 86%
F5 29 86%

F6 1 0%

Un-‐intui@ve drop behavior
of router “X”
100GE
F1 150G F1 – F5
F2 50G F6
100GE
700G
F3 200G
F4 100G Intui've (loss)? Observed 2 (router X)

F5 200G
F1 21 86% 25 82%
F6 1G
F2 7 86% 8 82%
F3 29 86% 22 88%
F4 14 86% 11 88%
•  Why losses in F6?
F5
•  Why unequal losses in F1-‐ 29 86% 33 83%
F6 F6 1 0% 0 52%
Un-‐intui@ve drop behavior
of router “Y”
100GE
F1 150G F1 – F5
F2 50G F6
100GE
700G
F3 200G
F4 100G Intui've (loss)? Observed 2 (router X) Observed 2 (router Y)

F5 200G
F1 21 86% 25 82% 26 83%
F6 1G
F2 7 86% 8 82% 9 83%
F3 29 86% 22 88% 22 89%

•  Why losses in F6? F4 14 86% 11 88% 11 89%
•  Why unequal losses in F1-‐ F5 29 86% 33 83% 33 84%
F6
•  No losses in F6 on router Y F6 1 0% 0 52% 1 0%
Know your hardware
300+ Gbps
fabric port
•  Router X -‐> CIOQ IF 1 (100GE)

•  Router Y -‐> VOQ F1 150G F1 – F5
PFE1
PFE4
F2 50G F6
•  Fabric port -‐>
Switch fabric
IF2 (100GE)
300Gbps F3 200G
PFE2
F4 100G
•  Fair-‐share fabric
F5 200G
scheduler
F6 10G PFE3
Router X – CIOQ -‐ behavior
•  Flow F1&F2 shares same buffer F1
IF 1 (100GE)
F1 – F5
(queue) on PFE1. Same for F3&F4
PFE1
PFE4
F6
@ PFE2 and for F5&F6 @ PFE3
F2
IF2 (100GE)
Switch fabric
•  There is 3 ingress PFE that want
F3
PFE2
to talk to PFE4 F4
–  Fabric Scheduling gives 100Gbps to F5
PFE3
each ingress PFE F6
offered From fabric @ PFE4 On egress interface
Gbps Gbps Fabric Loss % (F_loss) Gbps Cumula've Loss %
F1
F2
150
50
75
25 } 100G
50%
50%
F3
F4
200
100
67
33
} 100G
67%
67%
F5
F6
200
10
95.2
4.8
} 100G
52%
52%
Router X – CIOQ -‐ behavior
•  Flows F1 – F5 (295Gbps) are queued F1
IF 1 (100GE)
F1 – F5
PFE1
PFE4
in OQ of IF1, and Tx @ 100Gbps. F2 F6
(65% loss – Egress interface loss; E-‐ IF2 (100GE)
Switch fabric
F3
loss)
PFE2
F4
•  Flows F6 (10Gbps) are queued in OQ F5
PFE3
of IF2, and tx @ 100Gbps. (0% loss) F6
Cumula've loss for F1-‐F5: F_loss + (1-‐F_loss)*E_loss

F1
F2
150
50
75
25 } 100G
50%
50%
25
8
82%
82%
F3
F4
200
100
67
33
} 100G
67%
67%
22
11
88%
88%
F5
F6
200
10
95.2
4.8
} 100G
52%
52%
33
4.8
83%
52%
CLI example
PFE 3
NPC3(eabu-sol-eng-be-mx480-2 vty)# sh cos halp fabric

queue-stats 4!
Des@na@on PFE
! (PFE4)
PFE index: 3 CCHIP 0 Low prio Queue: 4!
Queued :!
Packets : 4734895792 62812 pps!
201 Gbps
Bytes : 5734975634075 25125000 Bps!
Transmitted :!
Packets : 0 31406 pps!
Bytes : 0 12562500 Bps!
Tail-dropped pkts : 0 31406 pps!
Tail-dropped bytes: 0 12562500 Bps! 50% loss
[...]!
Router Y – VOQ -‐ behavior
IF 1 (100GE)
F1 – F5
•  Flow F1&F2 shares same buffer (queue) on
F1
PFE1
PFE4
PFE1. Same for F3&F4 @ PFE2 and for F5 @ F2 F6
IF2 (100GE)
PFE3.
Switch fabric
F3
PFE2
•  Flow F6 has separate buffer on PFE 3. F4
•  There is 3 ingress PFE that want to talk to F5

egress interface IF1 (100GE)
PFE3
F6
–  Fabric Scheduling gives 33Gbps to each
ingress PFE for Egress Interface IF1 VOQ
F1
F2
150
50
25
8 } 33G
83%
83%
F3
F4
200
100
22
11 } 33G
89%
89%
F5 200 33 } 33G 83%
F6 10 10 } 100G 0%
Router Y – VOQ -‐ behavior
IF 1 (100GE)
F1 F1 – F5
•  There is 1 ingress PFE that want to talk to
PFE1
PFE4
F2 F6
egress interface IF2 (100GE) IF2 (100GE)
Switch fabric
F3
–  Fabric Scheduling gives 100Gbps to only one
PFE2
ingress PFE (PFE3) for Egress Interface IF2 VOQ F4
•  F6 do not consume it’s share in full. Only F5
PFE3
10Gbps. F6
Cumula've loss for F1-‐F5: F_loss

F1
F2
150
50
25
8 } 33G
83%
83%
25
8
83%
83%
F3
F4
200
100
22
11 } 33G
89%
89%
22
11
89%
89%
F5 200 33 } 33G 83% 33 83%
F6 10 10 } 100G 0% 10 0%
CLI example
Ingress PFE 3 VOQ of egress IF
(200Gbps toward egress interface) (100GE)
SNGFPC1(Thorax-re0 vty)# debug cos halp qlen tq 3 voq 2048!

<snip>!
!
VOQ| AQID| qlen| qlenold| tabw| ntabw|
maxrate(Mbps)| DBB Time(us)|!
==============================================================
==================!
560| 29771| 2752| 2304| 64| 14| 33Gbps drain-‐rate
1613| 111813|!
==============================================================
==================!
!
Number of Samples Collected = 1!
Parm | Min| Avg| Max|!
===============================!
qlen | 2752| 2752| 2752|!
tabw | 64| 64| 64|!
ntabw | 14| 14| 14|!
qdrain| 33000| 33000| 33000|!
FreePg| 259387| 259387| 259387|!
UM | 0| 0| 0|!
Accept your router personality
•  Behavior is clear now J -‐ caused
by Fabric scheduler
–  Non of vendor (AFAIK) allows for
Fabric scheduler tuning. F1 150G
PFE1
F4’ 33G
–  Have to live with it as it is. F2 30G
F3 200G
PFE2
Mi@ga@on: Load PFE fairly F4’’ 33G
•  you get fair results among all flows F5 200G
PFE3
F4’’’ 33G
•  Other goodies behind – blast F6 1G
radius
ROUTER BEHAVIOR CASE STUDY -‐
LATENCY
Intui@ve latency
100GE
F1 200G F1 – F3 400Gbps -‐-‐> 100Gbps
Buffer 100% -‐> 100ms latency
400G
F2 100G And 75% losses
F3 100G
Observed 1
Expected (loss)? Observed 2 (router Y)
(router X)
ms ms ms
F1 100 200 100
F2 100 200 100
F3 100 100 100

Know your hardware
300+ Gbps
fabric port
•  Router X -‐> CIOQ
IF 1 (100GE)
–  100ms VoQ before F1 200G F1 – F3
PFE1
PFE4
fabric F2 100G
–  100ms OQ
Switch fabric
F3 100G
PFE2
•  Router Y -‐> VOQ
•  100ms VoQ before fabric
PFE3
•  Fabric port -‐>
300Gbps
Router X -‐ CIOQ
•  Flow F1&F2 shares same VoQ buffer
(queue) on PFE1. Flow F3 is alone on F1-‐F3
PFE1
PFE4
F1
PFE2 F2
Switch fabric
IF 1
•  There is 2 ingress PFE that want to F3 (100GE)
PFE2
talk to PFE4
–  Fabric Scheduling guarantee 150Gbps
PFE3
to each ingress PFE
•  Flows F1-‐F2 (300Gbps) are queued
in VOQ of PFE1, and Tx @ 200Gbps
(150Gbps + le•over).
Latency is 100ms
offered Fabric component Egress Interface Total
Gbps ms
F1 200 100 (33% loss)
F2 100 100 (33% loss)
F3 100
Router X -‐ CIOQ
•  Flows F3 (100Gbps) are queued
F1-‐F3
PFE1
PFE4
F1
in VOQ of PFE1, and Tx @ F2
Switch fabric
IF 1
100Gbps à no buffering (100GE)
F3
PFE2
•  Flows F1-‐F3 (300Gbps) are
queued in OQ of IF1, and Tx @
PFE3
100Gbps.
Egress interface latency is 100ms

Gbps ms Ms ms
F1 200 100 (33% loss) 100 (66% loss) 200 (77% loss)
F2 100 100 (33% loss) 100 (66% loss) 200 (77% loss)
F3 100 0 (0% loss) 100 (66% loss) 100 (66%)
Router Y -‐ VOQ
•  Flow F1&F2 shares same F1-‐F3
PFE1
PFE4
F1
VoQ buffer (queue) on PFE1. F2
Switch fabric
IF 1
Flow F3 is alone on PFE2 F3 (100GE)
PFE2
•  There is 2 ingress PFE that
PFE3
want to talk to PFE4
–  Fabric Scheduling guarantee
50Gbps to each ingress PFE

Gbps ms ms sm
F1 200 100
F2 100 100
F3 100 100
Router Y -‐ VOQ
•  Flows F1-‐F2 (300Gbps) are F1-‐F3
PFE1
PFE4
F1
queued in VOQ of IF1, and Tx F2
@ 50Gbps.
Switch fabric
IF 1
F3 (100GE)
Latency is 100ms
PFE2
•  Flows F3 (100Gbps) are
PFE3
queued in VOQ of IF1, and Tx
@ 50Gbps
Latency is 100ms

Gbps ms ms sm
F1 200 100 (83% loss) 0 100 (83% loss)
F2 100 100 (83% loss) 0 100 (83% loss)
F3 100 100 (50% loss) 0 100 (50% loss)
Other surprising behavior – router W
100GE
PFE1 F1, F4, F6
PFE4
F1 100G
F2 100G F2, F5, F7
F3 100G F3
Switch fabric 100GE
F4 100G
PFE2
Observed on egress
Gbps Loss %
F5 100G F1
PFE3
33 67%
F6 100G F2 50 50%
F7 100G
F3 100 0%
F4 33 67%
•  All traffic is BE F5 25 75%
F6 33 67%
F7 25 75%
Homework
•  What Queuing architecture router W is?
•  Explain behavior.
•  Answers: rafal@juniper.net
–  Deadline – 11pm today.
Summary
•  Know your router anatomy
•  System queuing architecture impacts power
consump@on and system scaling capabili@es.
•  System queuing architecture impact residency-‐
@me
•  System queuing architecture may be a reasoned
for non-‐intui@ve traffic loss paPern.
–  (Re)Assign ports to roles smart way.
–  Trying to solve of non-‐exis@ng problem cost @me and
headache of wri@ng a incident report – avoid it.
BACKUP SLIDES.
From building blocks to centralized
router
•  Single switching element
•  No Switch Fabric
•  N x N Interfaces
N x Rx
N x Tx
1 x Rx
1 x Tx
•  Interfaces may have different speeds

•  Memory used to build egress
interface queues
•  Different memory op@ons
–  On-‐chip shared by mux and de-‐mux
•  Very fast (SRAM) – ~10Tbps+
•  Small and costly (10’s MB)
–  Off-‐chip shared by mux and de-‐mux
•  Deep queues/buffers (GB)
•  Slower (DRAM) – ~1Tbps
N x Rx
N x Tx
–  Off-‐chip memory limits PFE

performance.
Combined Input Output Queuing
IF0 Tx
IF1 Tx
Port P1 Rx Port P1 Tx Pn RX
P1 OQ
IFn Tx
P2 OQ
Port Pn Rx Port Pn Tx
Pn OQ
IQ requairments for loss-‐less:

Fabric Port speed >> Σ (ASIC egress interfaces)
OQ
•  Fabric switch is not a
really switch here.
•  Need very high speed (N
P1 OQ
Port P2 Rx
P2 OQ
Port P2 Tx x ) fan-‐in to buffer.
–  If switch port is 600Gbps,
and we have 100 ports à
–  60T of raw bandwidth into
Port Pn Rx
Pn OQ
Port Pn Tx
buffers !
•  100% efficiency
N x Port Speed
N x N x Port Speed
•  Good on paper only
–  extremely expensive
–  Bound by technology
Buffer once – per egress interface VOQ
Theory vs. prac@ce
Data in
Egress Interface busy

Data in VOQ
•  In stat-‐mux system it is RQ shallow
output
unpredictable when egress buffer
interface becomes free.

•  Grant signaling delay affect
deficiency GR
•  Need for shallow buffer a•er
Egress Interface
fabric (egress PFE) to DATA
busy
compensate delay.
–  Need just ~10 usec.
•  Similar to CIOQ but:

–  Fabric flow-‐control ensure
that OQ never overflow.
–  Try to keep IQ always full,
never empty (if data are in
VOQ)
–  Egress interface free/busy Data in
shallow
indirectly controls fabric output
flow-‐control buffer
Buffer once – per egress interface VOQ
Theory vs. prac@ce
Data in
shallow

Data in VOQ
•  In stat-‐mux system it is RQ output
buffer
unpredictable when egress
interface becomes free.
•  Grant signaling delay affect
deficiency GR
•  Need for shallow buffer a•er
Egress Interface
fabric (egress PFE) to DATA
busy
compensate delay.
–  Need just ~10 usec.
•  Similar to CIOQ but:

–  Fabric flow-‐control ensure
that OQ never overflow.
–  Try to keep IQ always full,
never empty (if data are in
VOQ)
–  Egress interface free/busy Data in
shallow
indirectly controls fabric output
flow-‐control buffer
Too shallow shallow output buffer
•  Latency

–  3 x fabric one-‐way latency is
worst case (RQ-‐GR-‐DATA) Data in
–  RQ scheduler on egress RQ shallow
output
–  GR scheduler on ingress buffer
Data in VOQ
•  RQ latency
–  Can’t be compensated
latency
loss
RQ
–  Sta@s@cally minor problem –
asynchronous. RQ could be GR
send while egress interface
Interface
Egress
handles other data
busy
•  RQ/GR scheduler -‐ Can’t be
compensated DATA
•  If shallow buffer < 2x latency –
inefficient egress IF u@liza@on
Egress Interface
busy
Impact on PFE design
PFE complexity
•  PFE for per-‐egress IF VOQ system != PFE for CIOQ
system
•  For system of 4k IF and 8 QoS Classes
–  VOQ PFE need support 32.000 queues
–  CIOQ PFE need 1.040 queues
•  If PFE support 400k queues
–  VOQ system can support 50k IF
–  CIOQ system can support 1.000.000’s IF (from queue
scaling perspec@ve only. Other limits apply)
•  CIOQ PFE has typically less queues but much
more of other func@onali@es.
PFE performance
•  To handle 4 x 100GE interfaces, PFE
need:
Memory –  CIOQ:
DRAM •  more then 3.6Tbps of PFE I/O.
•  All packet goes to and from memory
twice – 4 memory accesses.
•  1600 (CIOQ) –  VOQ
or
•  800 (VOQ)
•  more then 2.4Tbps (25% less)of PFE.
•  All packet goes to and from memory
once – 2 memory accesses.
Mem I/O –  Memory I/O BW need to be
oversized -‐ even bePer reduc@on
(~30%)
External IF I/O
400 400 –  Each memory access causes latency

Fabric I/O
PFE
•  Less I/O == saved gates
–  Have more VOQ (bigger system), OR
400 400
–  lower cost and power, OR
–  higher performance, OR
CoS vs. Queuing
Queueing vs. QoS
•  Queuing as discussed so fare (ingress PFE)
–  manage conges@on/overload Class 0
(4 classes)
–  Assumes all traffic is same class Class 1 Class 0
CoS
Class 1 egress PFE 0
•  CoS
egress PFE 1
Class n
Class n egress PFE n
–  Traffic has different classes

–  Each class need other treatment
no CoS
egress 0
•  CoS + Queueing = QoS egress 1
egress n
–  Scheduler(s) take into account 2 things
•  Class of traffic IQ VoQ
•  Source instance (e.g. ingress PFE)
–  What was 1 Queue becomes set of parallel queues.
–  Classifier needed -‐> put traffic to this or other queue (out of
scope)
PFE complexity
•  PFE for per-‐egress IF VOQ system != PFE for CIOQ
system
•  For system of 4k IF and 8 QoS Classes
–  VOQ PFE need support 32.000 queues
–  CIOQ PFE need 1.040 queues
•  If PFE support 400k queues
–  VOQ system can support 50k IF
–  CIOQ system can support 1.000.000’s IF (from queue
scaling perspec@ve only. Other limits apply)
•  CIOQ PFE has typically less queues but much
more of other func@onali@es.
CIOQ-‐ Fabric SWITCH
FABRIC
•  CIOQ
•  Fabric VOQ
–  Flow-‐Controll
•  per Fabric Egress
•  By Request – Grant protocols.
•  Do not depends on Egress
interfaces and OQ state. Hi-‐Priority
Queue
–  2 Classes Hi-‐/Low-‐ priority Ingress PFE1
Low-‐priority
•  classifica@on Queue
–  Scheduling Hi-‐Priority
•  ingress PFE fairness 1st Queue
Ingress PFE2 root
•  CIR-‐bound Priority scheduling. Low-‐priority
Queue
–  Delay Bandwidth Buffer:
Hi-‐Priority
•  Scheduler logic seats on egress Queue
PFE; Memory is on ingress side of Low-‐priority

Ingress PFEn
fabric. Queue
CIOQ – Egress Interface
•  CIOQ
Hi-‐Priority
Queue
Ingress PFE1
Low-‐priority
Queue
•  Egress (logical) Interface Queues Hi-‐Priority

Queue
Low-‐priority
Ingress PFE2 root
–  8 Queues per (logical) Interface

Queue
Hi-‐Priority
Queue
Ingress PFEn
–  10M queues per system Low-‐priority

Queue
–  Scheduling
•  5 priori@es level, 4 RED profiles per
queue. Configurable.
•  2 or 4 scheduling hierarchy level w/ Queue 0
priority propaga@on
–  Delay Bandwidth Buffer: ______
Queue 1
Logical
interface
logical
•  Scheduler logic and Memory seats

Queue 2 interface set
Logical
interface
Egress
on egress PFE Queue 3

Logical
Interface 1
classification
interface
logical
Queue 4 interface set
Logical
interface
Queue 5
Queue 6
Queue 7
SWITCH
FABRIC
VOQ – integrated scheduler VOQ_Class_0
VOQ_Class_1
•  Flow-‐Control VOQ_Class_2
–  per egress (logical) interface

VOQ_Class_3
–  By Request – Grant protocols. Ingress PFE1
–  Depends on Egress interfaces OQs (Output Queues) VOQ_Class_4
state.
VOQ_Class_5
•  Queues
•  8 per egress (logical) interface VOQ_Class_6
Logical
•  390.000 per system (HW limit) VOQ_Class_7
interface
•  Scheduling Logical
interface
Egress
VOQ_Class_0
–  4 priori@es level, 4 RED profiles per queue. Logical
Interface
Configurable. VOQ_Class_1
interface
–  2 or 3 scheduling hierarchy level w/ priority Logical

interface
propaga@on VOQ_Class_2
•  Delay Bandwidth Buffer: VOQ_Class_3
–  Shared memory Ingress PFE2

VOQ_Class_4
–  100ms upper bound Ingress PFE3
–  40ms worst case. VOQ_Class_5
•  Scheduler logic seats on egress PFE; Memory is

Ingress PFEn
VOQ_Class_6
on ingress side of fabric

VOQ_Class_7

Packet-Buffering Inside Routers

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Packet-Buffering Inside Routers

Hochgeladen von

Copyright:

Verfügbare Formate

Strategies

of packet buﬀering

Port u'liza'on 100%, port

Defect (bug) or expected behavior ?

• QoS goal -­‐ provide diﬀeren@al

This talk is not on QoS.

PFE – a CPU, NPU or ASIC that process packet

• One to many

• 1 Rx to 1 Tx @ same

• N Rx to 1 Tx @ same @me

• Input Queuing – IQ

• Virtual Output Queuing -­‐ VOQ

desired port traﬃc to

get 99%+ eﬃciency

RQ – Each has data size

– when egress Fabric port is

– E.g. RoundRobin or fair-­‐

– Extension to IQ

– Ingress PFE can send

requests to mul@ple

grants to mul@ple

grants to mul@ple

• Mul@stage • hiSpeed (fab) to low speed port

– Max: Σ (VOQ size, OQ size)

Pn VOQ IFn OQ share) from given egress

Ingress PFE egress PFE

• Packet received from

Ingress PFE egress PFE

Ingress PFE egress PFE

• Packet received from

Fabric is immediately send

out by egress interface

• Fabric queuing and flow-control depends on egress interface only.

F4 100G Intui've (loss)? Observed 2 (router X) Observed 2 (router Y)

Gbps Loss % Gbps Loss % Gbps Loss %

F4 100G Intui've (loss)? Observed 2 (router X)

Gbps Loss % Gbps Loss % Gbps Loss %

F4 100G Intui've (loss)? Observed 2 (router X) Observed 2 (router Y)

Gbps Loss % Gbps Loss % Gbps Loss %

F3 29 86% 22 88% 22 89%

• Router X -­‐> CIOQ IF 1 (100GE)

– Fabric Scheduling gives 100Gbps to F5

oﬀered From fabric @ PFE4 On egress interface

Gbps Gbps Fabric Loss % (F_loss) Gbps Cumula've Loss %

(65% loss – Egress interface loss; E-­‐ IF2 (100GE)

• Flows F6 (10Gbps) are queued in OQ F5

Cumula've loss for F1-­‐F5: F_loss + (1-­‐F_loss)*E_loss

Gbps Gbps Fabric Loss % (F_loss) Gbps Cumula've Loss %

NPC3(eabu-sol-eng-be-mx480-2 vty)# sh cos halp fabric

• There is 3 ingress PFE that want to talk to F5

Gbps Gbps Fabric Loss % (F_loss) Gbps Cumula've Loss %

F5 200 33 } 33G 83%

• F6 do not consume it’s share in full. Only F5

Cumula've loss for F1-­‐F5: F_loss

Gbps Gbps Fabric Loss % (F_loss) Gbps Cumula've Loss %

F5 200 33 } 33G 83% 33 83%

SNGFPC1(Thorax-re0 vty)# debug cos halp qlen tq 3 voq 2048!

• you get fair results among all ﬂows F5 200G

F2 100G And 75% losses

F1 100 200 100

F2 100 200 100

F3 100 100 100

•  QoS goal -‐ provide diﬀeren@al

•  One to many

•  1 Rx to 1 Tx @ same

•  N Rx to 1 Tx @ same @me

•  Input Queuing – IQ

•  Virtual Output Queuing -‐ VOQ

RQ –  Each has data size

–  E.g. RoundRobin or fair-‐

–  Extension to IQ

–  Ingress PFE can send

•  Mul@stage •  hiSpeed (fab) to low speed port

–  Max: Σ (VOQ size, OQ size)

•  Packet received from

•  Packet received from

•  Fabric queuing and flow-control depends on egress interface only.

•  Router X -‐> CIOQ IF 1 (100GE)

–  Fabric Scheduling gives 100Gbps to F5

(65% loss – Egress interface loss; E-‐ IF2 (100GE)

•  Flows F6 (10Gbps) are queued in OQ F5

Cumula've loss for F1-‐F5: F_loss + (1-‐F_loss)*E_loss

•  There is 3 ingress PFE that want to talk to F5

•  F6 do not consume it’s share in full. Only F5

Cumula've loss for F1-‐F5: F_loss

•  you get fair results among all ﬂows F5 200G

•  All traﬃc is BE F5 25 75%

•  Interfaces may have diﬀerent speeds

–  Oﬀ-‐chip memory limits PFE

IQ requairments for loss-‐less:

400 400 –  Each memory access causes latency

–  manage conges@on/overload Class 0

–  Traﬃc has diﬀerent classes

•  CoS + Queueing = QoS egress 1

PFE; Memory is on ingress side of Low-‐priority

•  Egress (logical) Interface Queues Hi-‐Priority

–  8 Queues per (logical) Interface

–  10M queues per system Low-‐priority

•  Scheduler logic and Memory seats

–  per egress (logical) interface

–  Depends on Egress interfaces OQs (Output Queues) VOQ_Class_4

–  2 or 3 scheduling hierarchy level w/ priority Logical

•  Delay Bandwidth Buﬀer: VOQ_Class_3

–  Shared memory Ingress PFE2

–  40ms worst case. VOQ_Class_5

•  Scheduler logic seats on egress PFE; Memory is