When simplicity becomes complex

On the road to a scalable and dynamic SURFnet7 network

SURFnet7 indepth
Wouter Huisman

What do we want from a network?

Scalable
Flexible
Cost efficient
Endusers service requirements

Bottomline is keep it simple

SURFnet6 network
Hamburg

Groningen1

Hamburg

Middenmeer1

Harlingen

Winschoten1

Leeuwarden

Assen1

Den Helder

Emmeloord

Beilen1

Dwingeloo1

Emmen1
Hoogeveen1

Subnetwork 4:
Blue Azur
Lelystad2

Meppel1
NLR

NLR
DLO

Alkmaar1

BT

ROADM

Amsterdam1

Haarlem1

Amsterdam2

Almere1

Lelystad1

Zwolle1

DAS3

Enschede1

Apeldoorn1

Leiden1

Mnster

SWOV
NBD

Hilversum1
Schiphol
Oegstgeest
Lisse

Subnetwork 3:
Red

Muenster1

3XLSOP

Breukelen1

Arnhem

Schiphol-Rijk

Subnetwork 1:
Green

TNO
Soesterberg

Zutphen1

Zoetermeer
LSOP Rtd

Wageningen1

ROADM

DenHaag
Rotterdam4

Amersfoort1

NNI
InHolland

Rotterdam1
Dordrecht1
Middelburg

Zierikzee

Bergen-op-Zoom

Nijmegen1

Ede

Nieuwegein1
Subnetwork 2:
Dark blue

Breda1

Den Bosch1

Venlo1

Eindhoven1
Vlissingen

ROADM

Krabbendijke

Venlo2

Roermond1

Subnetwork 5:
Grey

Sittard1

Tilburg1
Maasbracht1

Heerlen1
Heerlen2

Brussel

Maastricht1

Maastricht7

Aaken

Aachen

Paris

3
Geneva (CERN)

Geneva

Collapsed IP core
Border Routers

Located in Ams1
and Ams2

SURFnet
AS1103

Core Routers

Customer
SURFnet SNE masters BGP in de praktijk

Transport layer

Juniper T-series: 4

Ciena OM5200: 20

Nortel MERS 8600: 20

Ciena OME6500: 100

Ciena CPL: 100

Nortel OME1010/1060: 350

What to optimize of SURFnet6?

And make SURFnet7 simple

SURFnet7:
Scalable network
Institute
Applica<on

Applica<on

Applica<on

Routed IP
(Layer 3)

2008
Next Generation
Ethernet

Service

Institute
Applica<on

Applica<on

Applica<on

Service

(Layer 2)

Bandwidth

13

2010

Bandwidth
Optisch Transport
(Layer 0/1)

Concepts in the SURFnet7 tender

Single technology end-2-end transport

Remove as many OEO as possible
Introduce 40G and 100G
Improve the BoD solution

Selected Ciena CES solution

5410, 3930, 5150
Based on PBB-TE

Ciena CESD solution

Ciena 5410
Core switch, 2 per main POP
Modular switch with 400Gb/s per slot

32-port 1G
4-port 10G
10-port10G
40-port 10G/48-port 1G
4-port 100G/2-port 40G

Pluggable optics

1G SX, LX, CWDM, DWDM

10G SR, LR, CWDM, DWDM
40G LR4 CFP
100G LR4 CFP

Ciena 5150

Pizza box with extension slots

2U height
48x 1G SFP
4x 10G XFP
Dual AC/DC power

Ciena 3930

Pizza box
1U height
8x 1G SFP
2x 1G/10G SFP+
Dual AC/DC power

Tiered Network Architecture

Tiered Network Architecture

Tiered Network Architecture

Tiered Network Architecture

Tiered Network Architecture

Tunable XFP & OTN XFP

5410 - CES

CPL DWDM OADM

Transponder

No transponder shelves required

Tunable DWDM XFP used
Save on power, footprint, OAM
~25% of the links require G.709

Scalability issue
Uptake lightpath services
was significant
Technology limitation
Bandwidth claimed for
100%, even though not
all used.
Single service per port

Statistical Multiplexing
EIR

CIR

a,b,c

b
c

26

a
b
c

QoS
2 Color marking (Green and Yellow)
8 levels with .1p priority
Management mapped to 7
Lightpath mapped to 5
IP services mapped to 3

Strict queueing, which guarantees that

Lightpaths suffer last from congestion

PBB-TE Encapsulation

Payload

Payload

Payload

Payload

SA

VID

C-VID

C-VID

DA

SA

S-VID

S-VID

DA

SA

SA

DA

DA

802.1

802.1q

802.1ad

I-SID
B-VID
B-SA
B-DA

802.1ah

28

PBB-TE Protection

Connection Oriented Ethernet

Primary & Backup Tunnels Monitored via 802.1ag CCMs

CCMs configurable for 10 msec intervals

MEP A
MEP C

Ingress PBB-TE
Edge Bridge

MEP B

Primary Tunnel
Protected Node

MEP D

Egress PBB-TE
Edge Bridge

29

 Node config of a tunnel, transits, services

Concept VS

Creating a tunnel
A
Startnode

C
B
Transit node

Transit node

Z
Endnode

virtual-switch create vs TRANSIT-1334!

pbt transit create pbt-transit TRANSIT-1334_503 parent-port 5/3!
pbt transit add pbt-transit TRANSIT-1334_503 class-element 1 bvid 1334!
vlaninterface
create vlan
1337pbt-transit
name TRANSIT-1337!
virtual-switch
attach
TRANSIT-1334_503 vs
vlan add vlan 1337 port 3.1!
TRANSIT-1334!
addpbt-transit
vlan 1337 port
to_Asd001A_5410_01T_l1!
pbt transitvlan
create
TRANSIT-1334_101
parent-port 1/1!
pbt transit add pbt-transit TRANSIT-1334_101 class-element 1 bvid 1334!
virtual-switch interface attach pbt-transit TRANSIT-1334_101 vs
TRANSIT-1334!

Creating a tunnel
A
Startnode

C
B
Transit node

Transit node

Z
Endnode

pbt tunnel-group create group DH5150_5410s tunnel-sync on!

pbt encap-tunnel create static-encap P_DH5150_5410s tunnel-group
DH5150_5410s pair-index 4 port to_Asd001A_5150_02T_l1 bvid 1333 destbridge-name Asd001A_5150_02 weight 6!
pbt decap-tunnel create static-decap P_DH5150_5410s tunnel-group
DH5150_5410s pair-index 4 port to_Asd001A_5150_02T_l1 bvid 1333 srcbridge-name Asd001A_5150_02!
cfm service create static-encap P_DH5150_5410s name PBT-1333 next-mepid
1333 level 2!
cfm service set service PBT-1333 ccm-interval 100ms alarm-time 0!
cfm service enable service PBT-1333!

Creating a service
A
Startnode

C
Z

Endnode
virtual-circuit pbt create static-vc 3669IP tunnel 1Hedr1_Asd_D egress-isid 73860 ingress-isid 73860!
virtual-circuit pbt set static-vc 3669IP retain-stag yes!
virtual-switch add reserved-vlan 150!
virtual-switch ethernet create vs 3669IP vc 3669IP reserved-vlan 150!
virtual-switch ethernet set vs 3669IP description "3669IP"!
virtual-switch ethernet set vs 3669IP encap-cos-policy fixed encap-fixed-dot1dpri 3!
! MTU size increased to 9190 for CN 5150 Hedr001A_5150_01 port 1.1!
port set port 1.1 max-frame-size 9190!
port set port 1.1 vs-ingress-filter off!
virtual-switch ethernet add vs 3669IP port 1.1!
port set port 1.1 acceptable-frame-type all untagged-data-vs 3669IP!
aggregation set port 1.1 agg-mode manual!
lldp set port 1.1 mode rx-only notification off!
virtual-switch ethernet set port 1.1 vs 3669IP encap-cos-policy fixed encap-fixed-dot1dpri 3!
traffic-profiling set port 1.1 mode standard-dot1dpri!
traffic-profiling standard-profile create port 1.1 profile 1 name 3669IP01 cir 0 cbs 16 pir 1000000 ebs 64!
traffic-profiling set port 1.1 nonconform-standard-profile 3669IP01!
traffic-profiling enable port 1.1!
cfm service create vs 3669IP name 3669IP next 1 level 4!
cfm service set service 3669IP alarm-priority 3!
cfm service set service 3669IP alarm-time 10000!
cfm service set service 3669IP reset-time 3000!
cfm service set service 3669IP ccm-interval 1s!
cfm service enable service 3669IP!
! CN 5150 Hedr001A_5150_01 port 1.1 forced to Enabled!
port enable port 1.1!

virtual-circuit pbt create static-vc 2013LP tunnel 0809prot egress-isid 1073041 ingress-isid 1073041
virtual-circuit pbt set static-vc 2013LP retain-stag yes
virtual-switch add reserved-vlan 151
virtual-switch ethernet create vs 2013LP vc 2013LP reserved-vlan 151
virtual-switch ethernet set vs 2013LP description "2013LP"
virtual-switch ethernet set vs 2013LP encap-cos-policy fixed encap-fixed-dot1dpri 5
virtual-switch l2-cft set vs 2013LP tunnel-method transparent
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol 802.1x disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lacp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lacp-marker disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol lldp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol oam disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol rstp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-cdp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-dtp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-pagp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-pvst disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-stp-uplink-fast disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-udld disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol cisco-vtp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol gvrp disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol vlan-bridge disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol all-bridges-block disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol bridge-block disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol garp-block disposition forward
virtual-switch l2-cft protocol add vs 2013LP ctrl-protocol gmrp disposition forward
virtual-switch l2-cft enable vs 2013LP
port set port 1 vs-ingress-filter off
virtual-switch ethernet add vs 2013LP port 1
port set port 1 acceptable-frame-type all untagged-data-vs 2013LP
rstp disable port 1
mstp disable port 1
aggregation set port 1 agg-mode manual
lldp set port 1 mode disable notification off
port set port 1 untagged-ctrl-vs 2013LP
virtual-switch ethernet set port 1 vs 2013LP encap-cos-policy fixed encap-fixed-dot1dpri 5
traffic-profiling set port 1 mode standard-dot1dpri
traffic-profiling standard-profile create port 1 profile 1 name 2013LP01 cir 0 cbs 16 pir 1000 ebs 64
traffic-profiling set port 1 nonconform-standard-profile 2013LP01
traffic-profiling enable port 1
cfm service create vs 2013LP name 2013LP next 2 level 4
cfm service set service 2013LP alarm-priority 3
cfm service set service 2013LP alarm-time 10000
cfm service set service 2013LP reset-time 3000
cfm service set service 2013LP ccm-interval 1s
cfm service set service 2013LP dmm-interval 1s
cfm delay send service 2013LP local-mepid 2 mepid 1 repeat 1
cfm service set service 2013LP lmm-interval 1s
cfm frame-loss send service 2013LP local-mepid 2 mepid 1 repeat 1

Service CFM
MEP
A
Startnode

MEP

C
B

Z
Endnode

Service CFM is added to each service

- Run from client sub port to client sub port
- Takes the same path as customer data
- Set at 1s interval, to allow hitless tunnel protec<on switches

ideal measurement point for availability calcula<ons

Monthly Reporting
Availability is based on Service CFM alarms
Time between raise and clear of the alarm is the down time of
the service

Volume reporting is based on the standard MIB entries

Per port and per service (subservice of a physical port) is
possible

Considering reporting of latency and jitter

Can we manage elephant flows?

Unpredictability of the dynamic nature of BoD
BoD tend to generate large flows (> 1Gb/s)
Can we rely on the NMS?

BW enforcement for tunnels and services

Layered structure of tunnels and services
Each tunnel has a bandwidth profile
Services are added to tunnels as long as tunnel
can accommodate

Link capacity
Tunnel prole
Service BW

Tunnel CIR: 2 node network

A

Future lightpaths
Exis<ng lightpaths

Consider:
10G Link between A and B
Filled with single tunnel and consuming 50% of CIR bandwidth
5G SLP is to be reserved for new install
Tunnel prole is changed to claim 100% CIR bandwidth
In mean <me a DLP is reserved claim 1G bandwidth on the exis<ng tunnel, and
consumes the bandwidth of the SLP service
End result: SLP can not claim its bandwidth on this link

SLP: sta<c lightpath

DLP: dynamic lightpath

Solution
DLP can only be transported across DLP tunnels
Claim Bandwidth in advance
Create dummy tunnels with required CIR for new
services
Or increase the CIR bandwidth of existing SLP
tunnels

2 elevator dilemma
Assume 2 elevators, both
with capacity for exact 10
people

2 elevator dilemma
Assume 2 elevators, both
with capacity for exact 10
people

Drop at least 1 person

LAG problem
Consider Lightpath flows of 6 Gb/s and of 5 Gb/s
A CES will autonomous decide, which LAG member to use
A LAG of 2 x 10G should be sufficient
But you have 50% chance on packet drop

Problem with LAGs with flows > 10% physical link speed

Solutions to the LAG issue

Use higher line rate instead of LAGs
Use 100G

Use LAGs, but be conscious not to put high bandwidth

flows on a LAG as you are loosing control

NMS implements Strict & Loose CAC flag

Strict CAC LAG
LAG with 1 member
Can accommodate any services
Loose CAC LAG
LAG with possibly more link members
Accommodates Lightpaths upto 1Gb/s And IP

wouter.huisman[at]surfnet.nl

www.surfnet.nl
+31 30 2 305 305
Creative Commons Attribution license:
http://creativecommons.org/licenses/by/3.0/