Implementation of a

new Optical Platform in

X-WiN
Peter Kaufmann/DFN
TERENA Network Architects Workshop
22. November 2012
Agenda

• Reasons & Requirements

• Results
• Status of Implementation

Page 2
Reasons & Requirements (1)
Why change?
• Capacity: Less than 50% of our lambdas are in use!
• Upgrade towards 40/100G only with new h/w
• Current equipment is approaching end of life (from 2006)
(investments into old h/w not usefull)

Most important
• Operational inflexibilities very annoying
 – as lambdas has
to be changed
– as more native lambda services are provided/added
– as transponders are dedicated/configured for each connection

Page 3
Reasons & Requirements (2)

Add flexibility
• Transparent lambdas between any core nodes (no OEO)
• Protected lambdas
• Easy and quick channel setup
– rapid service delivery
–
flexible use of transponders
– ability to reroute lambdas
easily in case of
serious link problems
• Optical switching facilities at fiber junctions
• Sub-Lambda switching facilities (OTN) (due to 10/100)

Page 4
Reasons & Requirements (3)

But also: Increase transport capacity

• compensate traffic growth
• native 100G/channel capability for our most demanding
users

Page 5
Reasons & Requirements (4)

New optical transmission system

• About 80 lambdas
• Support for native 10G & 100G services
• Transparent lambdas between any core nodes (~1000km reach)
• ROADM: directionless & colorless (not fully contentionless, no flexgrid),
integrated OTN-Switching

• Migration philosophy
 – On top of current fiber infrastructure (only

few add. fibers)
– 100G transponders where needed, when needed
– Smooth migration, no operational „shut down“

Seite 6
Results
• Call for tender (non open, 2 stages): November 2011
– After stage one, reduction to „some serious“ participants
• Acceptance of tender: 10. May 2012
• Winner
– ECI Telecom, http://www.ecitele.com
– Israelien company, founded 1961
– about 2500 employes
– New player in „NREN environment“
• Contract & order for migration: 14. May 2012

• Product family „Apollo“

– Optimized Multilayer Transport (OMLT)
– integrated DWDM-/Switching-equipment
– DFN: OPT9624 at all core nodes (plus OPT9608 for some clients)
Seite 7
Results: ECI OPT96xx „Apollo“

 24/48 universal I/O slots

 Tbit/s Universal Switch:
OPT9624/48 for Metro ODU X-connect/Packet Switching
Core and Core/Regional/LH
 Flexible configuration:
Photonics, ODU-XC, Packet switching

 8 universal I/O slots

OPT9608 for Metro Edge  Flexible configuration: standalone WDM,
100G MPLS switching capacity

 4 universal I/O slots

OPT9604 for Metro Edge
 50G MPLS switching capacity

OPT9603 for Metro access  OPT9603 - 2U height with 3 universal slots

and In-Line amplifier
 Artemis: passive cages
Artemis

Quelle: ECI Telecom

Seite 8
Results: Fiber set-up
• High Capacity
– 88 Lambdas per Fiber
– Bandwidth up to 100 Gbit/s per Lambda

• Flexible Routing of Connections

– Reconfigurable Optical Add-Drop Multiplexer (ROADM)
– Colourless/Directionless Add-Drop
– Tunable Transponder

• Switching Functions
– Optional: 1Tbit/s Switching-Fabric per Node
– Currently: Usage as ODU Cross-Connect
– Future: In addition usage as MPLS- and Ethernet-Switch

Seite 9
 Capacities in Generations of WiN
Bandwidth in Core of WiN (Wissenschaftsnetzes):

X-WiN 2012: 8.800 Gbit/s

X-WiN 2006: 400 Gbit/s
G-WiN 2000: 10 Gbit/s
B-WiN 1995: 0,622 Gbit/s

Seite 10
Results: Fiber set-up

Amplifier
• Raman amplifiers compensate excessive attenuation on
long spans
• New: Mixture of Raman, EDFA
• Requires additional maintenance procedures

Redundancy
• Customer connection: Redundant fiber with two Core Nodes
(nearly all customers, 77% had not even 1 second interruption in 2011)
• Further extension of redundant fiber connectivity

Seite 11
Results: ROADM
• High Capacity
– 88 Lambdas per Fiber
– Bandwidth up to 100 Gbit/s per Lambda

• Flexible Routing of Connections

– Reconfigurable Optical Add-Drop Multiplexer (ROADM)
– Colourless/Directionless Add-Drop
– Tunable Transponder

• Switching Functions
– Optional: 1Tbit/s Switching-Fabric per Node
– Currently: Usage as ODU Cross-Connect
– Future: In addition usage as MPLS- and Ethernet-Switch

Seite 12
 Results: ROADM

ROADM functionality
• Colourless: Flexible A/D on any available Lambda
• Directionless: Flexible mapping on any required Direction
(But still uni-directional use of each part of fiber pair)
• Contentionless: Blocking-free switching of same colours, requires add. WSS
(Not that important for DFN, partially possible, not 100%)
• Flexgrid/Super Channels: Not required for DFN use cases during life time of gear

• Each ROADM configuered with at least: 4 degrees + 2 A/D blocks,

extension to 9 degrees possible
• OTN-Switch complements sub-lambda switching done in routers
• Restoration by ROADM: Switch time 1sec

Seite 13
ROADM: Optical Architecture

Source: ECI Telecom

Page 14
ROADM: L1 Service Cards

Client: SFP OTN XFP

16 x SFPs SFP Mapper
Multi service 10G STM-1/4/16, OC3/12/48 2 lines
AoC10 SFP
OTU-2/2e
Muxponder or AoC FC1/2/4, GbE,
SDI, HD-SDI, DVB-ASI
SFP
SFP XFP
OTU1

Client: XFP XFP

Multi service 10G 10G LAN,
OTN
Mapper
2 lines
double TR10_4 STM64/OC192, OTU-2/2e
FC8/10,
Transponder OTU-2/2e XFP XFP

Client:
Multi service 40G 10G LAN,
XFP OTN
Mapper
XFP
RZ-DQPSK CMB40 STM64/OC192,
XFP
QPSK
FC8/10, Line
Muxponder OTU-2/2e XFP OTU-3e

OTN
Mapper

OTU3 Regenerator REG40 QPSK QPSK

Line Line
OTU-3e OTU-3e

OTN
Mapper
Transponder/ QPSK QPSK
TR100
Muxponder 100G Line
100GbE
Line
OTU4

Source: ECI Telecom

Page 15
ECI: ROADM/Service-Cards
North

West East

Local A/D Without Fabric: L1 Service Cards

AoC10: 16*SFP -> 2*XFP-OTU2
TR10_4: 2*XFP -> 2*XFP-OTU2/2e
TR100: 1*CFP -> 1*CFP-OTU4 (later)

With Fabric: L1 Fabric Cards

FIO10_5: 5*XFP -> Fabric
FIO100: 1*CFP -> Fabric (later)
Fabric
FIO10_5: 5*XFP -> Fabric (later)
FIOMR_16: 16*SFP -> Fabric

Client
page 16
Results: Switching
• High Capacity
– 88 Lambdas per Fiber
– Bandwidth up to 100 Gbit/s per Lambda

• Flexible Routing of Connections

– Reconfigurable Optical Add-Drop Multiplexer (ROADM)
– Colourless/Directionless Add-Drop
– Tunable Transponder

• Switching Functions
– Optional: 1Tbit/s Switching-Fabric per Node
– Currently: Usage as ODU Cross-Connect
– Future: In addition usage as MPLS- and Ethernet-Switch

Seite 17
Optical Transport Network (OTN)

Source: Alcatel-Lucent Seite 18

OTN-Hierarchy

Source: Alcatel-Lucent page 19

 Switching: OTN-BB of X-WiN
KIE
• Operation of 1Tbit/s-Fabric (blue-red) DKR ROS
GRE
AWI DES
– Initially at 14 core nodes HAM
EWE
– Optimized with current network FFO SLU
BRE TUB
ENS PEP
structure and latency HAN ZEU
BIE POT HUB
MUE ZIB
– If needed: extensible PAD
BRA MAG ADH

DUI GOE
DOR
KAS
• OTN-Backbone (yellow) FZJ WUP
MAR
LEI DRE

AAC BIR JEN

GIE CHE
– Start: 2-3 OTU2-Connections per link BON
FRA ILM
– Future: Extension towards OTU3/4 WUE BAY
GSI ERL
ESF
SAA KAI
REG
• Access from User/Simple Core FZK
HEI

STB
KEH FHM
Node to Fabric STU
GAR
– All interfaces use OTH-framing BAS

– Single or redundant OTU2e-connection to

next one/two core nodes with fabric

Seite 20
Switching: L1 Fabric Cards

16 x SFPs SFP OTN

STM-1/4/16, OC3/12/48 SFP Mapper
Low rate client 1Tbps
FIOMR_16 FC1/2/4 SFP
ODU-XC
interface GBE, SFP
SDI,HD-SDI, DVB-ASI SFP
OTU1

XFP
Multi service 5 x XFPs
10G LAN, XFP
OTN
Mapper
1Tbps
10G client line FIO10_5 STM64/OC192, XFP
ODU-XC
FC8/10, XFP
Interface OTU-2/2e XFP

OTN
Mapper
40G RZ-DQPSK QPSK
1Tbps
FIO40 ODU-XC
line card Line
OTU-3e

OTN

100G PM-QPSK Mapper

1Tbps
FIO100 QPSK
ODU-XC
line card Line
OTU-4

Source: ECI Telecom

Page 21
ECI: ROADM/Fabric-Cards
North

West East

Local A/D Without Fabric: L1 Service Cards

AoC10: 16*SFP -> 2*XFP-OTU2
TR10_4: 2*XFP -> 2*XFP-OTU2/2e
TR100: 1*CFP -> 1*CFP-OTU4 (later)

With Fabric: L1 Fabric Cards

FIO10_5: 5*XFP -> Fabric
FIO100: 1*CFP -> Fabric (later)
Fabric

FIOMR_16: 16*SFP -> Fabric

FIO10_5: 5*XFP -> Fabric (later)

Client
page 22
Universal Switch Fabric

 The fabric can support ODU-XC,

MPLS switch or any mix of both
(depending on the service cards 10GbE
/ STM1-64 /
ODU-XC
(ODU-0-4, Flex)
installed in the system). FC10 / OTU-2
OTN OTN
10G

STM1 / 4 / 16 /
 Scalable to 4Tbit and future FC1 / 2 / 4 / GbE 40G
OTN OTN
16Tbit switching capacity (in
multi-shelf Architecture) 100GbE 100G
OTN OTN

 ODUk and ODU-Flex XC

100GbE
100G
 Transparency of timing and OH; Packet Packet
Each ODUk container carries its 10GbE 10G
own timing Packet Packet

1GbE 100G
 Each line card is configured to Packet Packet
work towards the fabric in either Packet
OTN mode or data mode Switching

Source: ECI Telecom Seite 23

Status: Migration Work

• Core Nodes (54)

– about 100 new chassis (power, etc.)

• Intermediate Line Amplifier: 45 Locations

• Fiber links (85)

– Total length about 10.500 km
– 33 parallel lines between central Core Nodes

• Lambda connections (167)

– Currently: Gigabit-Ethernet and 10-Gigabit-Ethernet
– With max. 18 hops and 2140 km fiber length

Page 24
Status of Implementation

Planning ECI- Provision and Operation ECI-

Components Components

Planning parallel Fiber

Infrastruktur Provision Parallel Fiber Infrastruktur

Planning
Implementation Location
Location-
Infrastructure
Infrastructure

Training
of DFN-
Employes

Preparation of Network Monitoring and

Adaption of Information Systems

05 06 07 08 09 10 11 12
Seite 25
Conclusion: Seen from Customer
• Increase of Bandwidth
– 100Gbit/s-Connections for VPN-service and DFNInternet (IP)

• Reduction of Provisioning Time for (new) Services

– Equipment is property of DFN: Better reserve of often required
components
– Flexibility: Components may used again for other connections

• New Switching Service

– Now: OTN, Mux of several 1Gb VPN over one 10Gb access
– Later: MPLS and Ethernet

• Reduction of Cost
– Components for 1Gbit/s- and 10Gbit/s-Connections much cheaper

Page 26
Questions ...?

?
? ?

Seite 27
ROADM – classical
• coloured
– For each „colour“ a specific port
per Add/Drop-Block
• Directed
– Exactly one outgoing direction erreichbar
per Add/Drop-Block
Add/

T
/
Drop
R

T
/
North

DWDM-
Knoten
Add/ Add/
Drop Drop
West East
T T T T
/ / / /
R X
R R R

Seite 28
ROADM – „modern“
• Colourless
– each „colour“ at any Add/Drop-Port
• Directionless
– each Ausgangsrichtung erreichbar at
any Add/Drop-Port

DWDM-
Knoten

Add/Drop
T T T T T
/ / / / /
R R R R R

Seite 29
Gigabit AoC / Client-Card

2xXFP OTU2 colored uplinks

16xSFP colored uplink

Source: ECI Telecom

Seite 30
10G Transponder / Fabric Card
2xXFP FC8/10,STM-64/OC-
2xXFP OTU2/2e/2f colored uplinks
192/10GBE/OTU2/2e B&W or
colored I/Fs

5xXFP FC8,STM-64/OC-192/10GBE/OTU2/2e B&W or colored I/Fs

Source: ECI Telecom Seite 31
Variants of „Apollo“

Pure WDM Application Metro Core / Regional / LH OMLT

 24 universal slots Photonic layer  4 slots for 1Tbps Photonic layer

 Photonics modules universal fabric
Low Line
 Service cards cards Rate 100G
TR100 Mux100
 L1 service cards  20 universal slots 10G
Line/Client
ODU-XC
Line
40G
 Photonics modules
TR40 Mux40 Client 10G
 Service cards 100G Line/Client

1GbE 100GbE
 L1 service cards L2/3 L2/3
TR10 AoC10 Packet
Switching
 L2/3 Data cards 10GbE
L2/3

Metro OMLT Application CESR Application

 2 slots for 100G Photonic layer  4 slots for 1Tbps Photonic layer
fabric cards universal fabric
1GbE
cards L2/3
 22 universal slots TR100 Mux100

 Photonics  20 universal slots 10GbE

L2/3
100GbE
L2/3
TR40 Mux40
modules  Service cards
Packet
 Service cards Switching
TR10 AoC10
 L2/3 Data cards
 L1 service cards 1GbE
L2/3
100GbE
L2/3
100G
1GbE 10GbE
Data
 L2/3 Data cards L2/3
Fabric L2/3 10GbE
L2/3

Source: ECI Telecom Seite 32

Service Restoration

3rd Restoration
Path

Main Path
E
A 1st Restoration F C
Path

G
2ndRestoration
Path
Restoration by ROADM. 
H Switch time 1sec 

S R
P O
A L A C
I D
T M

Seite 33
 Deutsches
Forschungsnetz
Bereitstellung Glasfaserstrecken
KIE
Glasfaserpaar
,(Bestand, DKR GRE
ROS
nicht parallel) AWI DES
HAM
Glasfaserpaar
(vorhanden) EWE
FFO SLU
Glasfaserpaar BRE TUB
(parallel, beauftragt) ENS ZEU
HAN
BIE POT HUB
Glasfaserpaar MUE ZIB
BRA MAG ADH
(parallel, Übergabe PAD
erfolgt) DUI GOE
BOC
Glasfaserpaar DOR
(parallel, Fertigstellung KAS
FZJ LEI DRE
bis Ende Oktober) WUP
BIR MAR JEN
AAC CHE
GIE
BON FRA ILM
WUE BAY
GSI

SAA KAI HEI ERL

REG
FZK STU
STB
KEH FHM

GAR
BAS
Seite 35
Stand Installation der Technik
KIE
Glasfaserpaar
,(Bestand, DKR GRE
nicht parallel) AWI DES ROS
HAM
Glasfaserpaar
(vorhanden) EWE
FFO SLU
Glasfaserpaar BRE TUB
(parallel, beauftragt) ENS HAN ZEU
BIE POT HUB
Kernnetzknoten MUE ZIB
(bereits aufgebaut) BRA MAG ADH
PAD
DUI GOE
Kernnetzknoten BOC
(geplant KW42) DOR
KAS
Kernnetzknoten FZJ LEI DRE
WUP
(geplant KW43) MAR
AAC BIR JEN CHE
Kernnetzknoten GIE
BON FRA ILM
(geplant KW44)
WUE BAY
Kernnetzknoten GSI
(geplant KW45)
SAA KAI HEI ERL
REG
FZK STU
STB
KEH FHM

GAR
BAS
Seite 36
Beteiligte Einrichtungen
• DFN Geschäftsstellen in Berlin und Stuttgart

• ECI Telecom
– beschäftigt zusätzlich Montagefirma für Aufbau DWDM-Technik (7 Teams)
– zusätzlich 6 eigene Teams für die Inbetriebnahme

• Glasfaserprovider
– stellen parallele Faserstrecken bereit und Stellflächen/DC-Versorgung an
ILA-Standorten
– insgesamt 10 Firmen

• Dimension Data
– Installationen an Kernnetzknoten (koordiniert zusätzlich Montagefirmen für
Schrankaufbau und Installation von Vorverkabelungen)
– Ersatzteillogistik (Zentrallager plus 10 regionale Depots)

• Gastgebende Einrichtungen an Kernnetzknoten

– zusätzlich Elektriker für die Installation von Stromversorgungen
Seite 37
Installationen an Kernnetzknoten
• 48V/DC Netzteile
– je 1-2 Chassis für USV/NS
– Kapazität je Gerät 4kW (zzgl. 2kW Spare)

• Fibre-Spooler/Cable Guides
– „saubere“ Führung von Glasfaserkabeln

• Passivgehäuse „Artemis“
– Aufnahme von passiven Modulen
(bspw. optische Splitter)

• Apollo 9624
– 1-4 Geräte je Standort
– Aufnahme der aktiven Komponenten
(Verstärker, ROADM, Line-/Fabric-Karten)

Seite 38
Installation am Standort JEN
• KNK Universität Jena (JEN)
– drei Kernnetzfasern
– zehn 10G-Verbindungen
– drei ECI OPT9624 Chassis

• Stellflächen
– Aufbau in nur einem Datenschrank,
Bestückung von Vorder- und Rückseite
– bisher wurden 3 Stellflächen genutzt

• Spannungsversorgung
– komplett redundant bzgl. Equipment und
Versorgungszweig
– kann zukünftig überwacht werden

Seite 39
Installation am Standort JEN
• Separates Chassis für ODU-XC
– drei 5-Port 10G-Karten
– je 2-3 OTU2 Trunks zu anderen XCs

Seite 40