Brkarc 3466

Engineering behind building a switch
BRKARC-3466
Carl Solder
BRKARC-3466 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Product Marketing
It Starts With…
CC – PRD - EC
Marketing Business Case The “Wish” List Engineering Study

Market Analysis All of the features, Resourcing
Technology Applicability functionality and Available Funds
Customer Requirements performance Timelines
Engineering Assessment characteristics required of Defines project
the product START!!
Switch Development Timeline
Mechanical Engineering
Hardware Functional Spec (HFS)
Hardware Functional Specification
Document Number
Author
<Code Name>
Product I.D.
Hardware Functional Specification
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent a diam felis, in lacinia erat. Vestibulum ante felis, imperdiet sed imperdiet sed, tristique auctor
nisl. Integer ullamcorper cursus velit, ac pretium erat consequat commodo. Phasellus id magna vitae lacus vestibulum placerat. Etiam urna justo, semper nec
adipiscing at, congue id sem. Vivamus sollicitudin dapibus dapibus. Nulla quis erat elit, ac sagittis libero. Integer eget lacus vel nulla feugiat lacinia. Vestibulum
lacinia pellentesque justo, quis ullamcorper tellus mollis non. Proin in quam ac ante consequat varius sed nec metus. Mauris posuere convallis dolor, quis
volutpat nulla accumsan id. Praesent posuere dictum justo sit amet auctor. Mauris a condimentum eros. Cras libero nisi, bibendum sit amet blandit non, ultricies
sed elit. Aliquam urna odio, facilisis ac vehicula eu, rutrum eget tortor. Nam eu rhoncus libero.
Nunc et felis ipsum, non consequat eros. Aliquam erat volutpat. Morbi tincidunt imperdiet lectus ac tristique. Nam commodo, lorem vitae gravida euismod, lorem
nibh ornare felis, ut fringilla massa erat vitae lacus. Proin vitae ipsum sit amet leo fringilla porta. Etiam mollis nulla id mauris porta faucibus. Donec pulvinar
posuere felis non sodales.
Mechanical Design (CAD)
Material Considerations
Initial design concept
Structural Design
Foundation for ID and ASIC
Structural Analysis – Shock Stress Distribution
Negative
Y Shock
Negative
Z Shock Positive
Y Shock
Positive Negative
Z Shock Positive X Shock
X Shock
Structural Analysis – Elastic Strain
After pressure
applied,… ability of
material to return to
its original shape
without incurring
fractures…
Structural Analysis – Seismic Simulation
Required Response Spectrum of

Bellcore
10
ZONE 1&2
ZONE 3
Acceleration (G)
ZONE 4
1
Help identify stress points in structure
Combination of random movements in X, Y and Z direction
0.1
0.1 1 10 100 Tested for and beyond required NEBS specification
Frequency (Hz)
Structural Analysis – Tilt Simulation
I o = mrOC2 + I c
H = r[sin(φ + 10 0 ) − sin φ ]
Impact on structure from tilt drop
1
mgH = Iω 2 Fully loaded chassis
2
Where
ω : Angular velocity
I : Inertial of mass
Total of 4 Impact drops
H : Height
m : total mass
- Front edge lifted up
g : Gravity acceleration
C.G : Center of gravity
- Left edge lifted up
- Rear edge lifted up
- Right edge lifted up
Output
10 Degrees Tilt - Stress/Strain Distribution
- Axial/Shear force on rivets
Mid Plane Design
Strength Analysis- Mid Plane Example
Mid Plane rigidity and strength is important

to avoid dislodging existing cards from the
mid plane connectors
Mid Plane Design
Strength Analysis- Mid Plane Example
At the point of connection, force needs to be applied

to insert connector (Linecard A)
Insertion force will bend Mid Plane
Might uncouple connector from existing installed

cards (Linecard B)
Next the Boards,… (Linecards)
Block Diagram
To Fabric Modules To Central Arbiters

EOBC
LC Arbitration
CPU Aggregator …
Fabric 2
40G SoC 40G SoC 40G SoC 40G SoC 40G SoC 40G SoC
Front Panel Ports
Static Flash
Memories ROM
SDRAM Power
RLDRAM PHY
CPU Clocking
Connectors Heat Sinks
LED’s PCB Layers
I/O Traces (nets)
Pins/Screws Vias
Joints Ejectors
Daughter DIMM’s
Boards FPGA’s
ASIC’s
Optics
Electronic CAD – Component Placement
ECAD team responsible for all

electronic component placement
ASIC’s
FPGA’s
RLDRAM
SSRAM
PHY’s
Clocks
CPU’s, etc
Tussle with Electrical and Mech

Eng teams on component
placement
Mechanical CAD (MCAD) MCAD team responsible for physical
component placement – power bricks,
I/O connectors, etc.
Build and Intermediate Data

Format (IDF) model for
Electrical Team
Yellow squares indicate “not to exceed”

height restrictions for components –
consideration for board placement
Thermal Profile Analysis
Thermal Analysis
Board Routing
A trace is a signal path between two components…
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
PORT PHY’S
TRACE 1
ASIC
TRACE 2
Board Routing
All trace lengths

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 MUST BE equal to
ensure functional
PORT PHY’S integrity of logic
circuits…
ASIC “Routing” of traces

a crucial aspect of
board layout
Board Routing
Do You Know?
How many traces (paths) on

Cisco’s new 40G Linecard?
Printed Circuit Board
Mechanically support and electrically connect

electronic components with conductive pathways
Each layer provides a base upon which a path (net)

links up silicon components
Vias (drill hole) exist in each layer to allow paths to

“cross” PCB layers
Do you know?
How many PCB layers make up Cisco’s new 40G

Linecard?
Final Design
Final Design
Final Layout Sent to Manufacturing
BRKARC-3466 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public
Final Board
Some Facts
26 PCB Layers
9,998 Nets
58,764 Vias
Industrial Design
Industrial Design
The story behind the development of

Cisco’s new 100G “CPAK” Optic
Industrial Design
Original design from

Lightwire and Cisco
Engineering
Industrial Design
Ground Breaking Product
BUT…
No specific look to
showcase technology
leadership
Industrial Design
Multiple Design Concepts drawn up
Industrial Design
Initial Design Concept handed to Engineering
Industrial Design
Engineering make changes based on thermal profile and testing
Design concepts lost in adjustments
The “Look” was turning ugly again…

Industrial Design
Industrial Design Revision kicks in again

Seek inspiration from cars that speak Speed and Precision
Industrial Design
Revised design based on car design inspiration

Speed – Directionality – Bold – Refinement
Industrial Design
New design needed to ensure it did not

impact thermal profile when plugged in cage
Industrial Design
Further Minor Adjustments NEW

Bigger Radius and Constant Width DESIGN
OLD
DESIGN
Industrial Design
OLD
DESIGN
DEEPER
CHAMFER
NEW
DESIGN
Industrial Design
SHARPER ANGLE for
easier cable release
NEW
DESIGN
OLD
DESIGN
Industrial Design
Color Block on Bottom

to indicate speed when
inserted up-side down.
Industrial Design
CLOSER MATCH FOR FRONT AND BACK
Industrial Design
Final Product
ASIC Engineering
ASIC Requirements
Marketing Engineering
FIB Table Size ECMP Paths LAG WCCP CoPP Policing

MAC Table Size IGMP v1/2/3 RACL CAPWAP L2 CoPP
IPv6 unicast/multicast PIM Snooping PACL EEE L3 CoPP
IPv4 unicast/multicast Netflow QoS 802.1Qbg FC/FCoE and more…
Adjacency Table Flow Masks PBR 802.1Qbh 1588 PTP
MPLS P/PE/TE EoMPLS RBACL 802.1Qaz 802.1ae
Latency VPLS VACL Adjustable MTU Port Security
Jumbo Frames GRE VACL Capture Byte Statistics SGT
Physical interfaces MPLSoGRE GOLD Packet Statistics AES Encryption
Logical Interfaces MVPN ELAM Broadcast Suppression VOQ
Trill SPAN VRF Lite Multicast Suppression QPPB
L2MP Paths ERSPAN Policing Unicast Suppression PFC
ASIC Functional Spec
ASIC Functional Specification
Document Number
Author
<Code Name>
Product I.D.
ASIC Functional Specification
Switch Architecture determines ASIC Architecture
Distributed ASIC
Architecture
Multiple Switch ASIC’s

Also known as SOC “Switch On a Chip”
Port ASIC’s
Consolidates Switch Functions
Fabric ASIC’s
Integrates all functions onto single ASIC
Forwarding ASIC’s
Forwarding :: Port :: Replication
Replication ASIC’s
ASIC Functions
Processing of packets Layer 2 Processing Fabric Buffers

both ingress and IPv4/IPv6 Processing Fabric Paths
egress MPLS Arbitration
VRF VoQ
Initial Packet Parse Netflow Serial Links
CRC Check Security ACL Flow Control
WRED Policing HOL Removal
Scheduling Statistics QoS
Remarking Tunnels
Packet Queues Recirculation
Packet Buffer Load Balancing
Packet Suppression Adjacencies
VLAN Translation
Board ASIC’s
10
10
5
Main Board ASIC/Memory
1
Components
4
2 1. CPU
6
2. Field Programmable Gate Array
(FPGA)
6
3. 40G MAC’s
4. Forwarding ASIC
7
8
5. Netflow
3 6. Security/QoS
7. Fabric ASIC
9
8. Fabric ASIC Interconnnect
3
9. Multicast/Replication ASIC
10. Adjacency
ASIC Forwarding Path
L3 Table
Egress Security
Parse ACLs
L2 Table VPN Ingress Security Adjacency Input / Output Fwd Update
Packet CAM Decision Statistics
ACLs Table Policing
Egress QoS ACL
Ingress QoS ACL
Note : There is no one correct forwarding path!!!

ASIC’s forwarding path dependant on required features
Define forwarding path for each packet

Code each functional block
Data Path and Port functions also architected

Component Decisions – CAM vs. TCAM
Content Addressable Ternary Content
Memory Addressable Memory
01001000 Lkup #1
01001110 01001101 Lkup #2
01001110 Lkup #3
1 01101010 1 01001010
2 01101011 2 010010XX Hit #1!
3 01001110 Hit! 3 01001XX0 Hit #3!
4 01001XXX Hit #2!
4 01101100
2 Result #1
4 Result #2
3 Result 3 Result #3
ASIC Block Diagram
SERDES
TCAM Adjacency MET SPAN VOQ’s Memories
TCAM Netflow
Recirculation L3
TCAM
Engine
TCAM Security
Buffer L2
TCAM QoS Memory CAM
Engine
Parser/Re-write
Ingress Egress
PCI
Port Logic Port Logic
ASIC Development Process
1 2 3
RTL
SYNTHESIS NETLIST
Register Transfer
RTL to Gates Gate Design
Language (VHDL)
6 5 4
PHOTOMASK PLACEMENT
Floor Plan
RTL to Gates Gates
7 8 9
Foundry
Device Test Packaging
Production
Verilog/VHDL Programming
Register Transfer Language
library IEEE;
use IEEE.std_logic_1164.all;
use IEEE.numeric_std.all; -- for the unsigned type
1 Gate approx equals 4 to 6 Transistor
entity COUNTER is
generic (
WIDTH : in natural := 32);
port (
RST : in std_logic;
CLK : in std_logic;
LOAD : in std_logic;
DATA : in std_logic_vector(WIDTH-1 downto 0);
Q : out std_logic_vector(WIDTH-1 downto 0));
end entity COUNTER;
architecture RTL of COUNTER is

signal CNT : unsigned(WIDTH-1 downto 0);
begin
process(RST, CLK) is
begin
if RST = '1' then
CNT <= (others => '0');
elsif rising_edge(CLK) then
if LOAD = '1' then
CNT <= unsigned(DATA); -- type is converted to
unsigned
else
CNT <= CNT + 1;
end if;
end if;
end process;
Q <= std_logic_vector(CNT); -- type is converted back to Rough ROT - 1 Gate equals 5 Transist
std_logic_vector
end architecture RTL;
Logic Transistors – An Intel Comparison
2000 2003 2006 2008 2010 2011
42M 220M 291M 781M 995M 2.3B

Transistors Transistors Transistors Transistors Transistors Transistors
Do You Know
How many Transistors in

Cisco’s Current Generation
Switching ASIC?
Gate Placement (Floor Planning)
Gate constructs assembled onto DIE space

Channel (Trace) definition
I/O and Power Planning
Clock Planning
Assembled like a jigsaw puzzle
Die Size / Yield Cost
Silicon Die Imperfections
Die Size / Yield Cost
http://commons.wikimedia.org/wiki/File:Wafer_die%27s_yield_model_%2810-20-40mm%29_-_Version_2_-_EN.png
ASIC Development
10μm 10um Intel 8008 (1971) Transistor Width
3um Intel 8088 (1975)
1.5um Intel 80286 (1982)
800nm Pentium P5 (1989)

µm/nm is often a
1μm 350nm Pentium II (1995)
reference to width of the
1um Intel 80386 (1985)
180nm Coppermine E (1999)
transistor
600nm PowerPC 601 (1994)
90nm VIA C7 (2002)
250nm AMD K6-2 (1998)
100nm 45nm Intel i7 (2008)
130nm PowerPC 7447 (2000)
Intel Ivy Bridge 22nm (2012)
65nm Intel Core2 (2006)
16nm (2014)
32nm Core i3 (2010)
Intel Sandy Bridge 32nm (2011) 11nm (2016)

10nm
1970 1980 1990 2000 2010 2020
Software Engineering
Software Architecture
Software Functional Spec (SFS)
Software Functional Specification
Document Number
Author
<Code Name>
Product I.D.
Software Functional Specification
Software vs. Firmware
Coding Process
Software Development Process
Do You Know?
How many features/lines of code

were typically introduced in a new
code release for the Catalyst 6500?
Hardware Test Engineering
Drop Test
Simulates drop of equipment onto a hard floor

Testing at different heights
Testing with different Chassis loads
Determine break points

Flat drops and angled drops tested
Stress fractures, rivet strength
Also test drop of equipment in packaging
All equipment should function after drop
Vibration Test
Shake on +ve/-ve X, Y and Z axis
Test for chassis/rivet stress
Components remain intact after test
Components are functional after test
Seismic Test
Test based on Loma Prieta Earthquake
7.2 Richter Scale
Load chassis at top of rack

allow for < 3” movement tolerance within
chassis structure
Confirm function while test in progress
Try to avoid
- Linecards popping out
- Linecards catapulting across room :-)
Humidity/Temperature
Device tested to NEBS Level 3 for both
Humidity/Temperature
Normal (controlled) operating environment

of 5C to 40C
Short term operation 5C to 49C
Up to 55C with adjusted altitude of 1829m

Cold start at 0C
Humidity levels 5% to 85%
Normally takes about one month to fully test

HALT Chamber
Testing beyond the standard NEBS Compliance
Heat / Altitude / Shock testing
Find where the product breaks under more

extreme conditions
Findings input into future product development

-20C +55C
-40C MDVT +80C
HALT
HALT Chamber
Shock Testing
Firing guns under test

platform bench
Loaded at different
angles – 30/60/90
Acoustics
EMI Chamber
Mechanical Design Verification Testing
Temperature Profile
RDT Chamber
• The Reliability Demonstration Test
(RDT) is Cisco’s approach to verifying
the stated reliability of a product prior
to production release.
• The reliability to be demonstrated is
the product’s MTBF (Mean Time
Between Failure).
• RDT replicates the end user operating
environment and application through
accelerated test time. It is expected
that all hardware features are
exercised in RDT.
• All new products including systems
and boards are subject to RDT.
Packaging Testing
Problem Comprehension Design Improvement Strategy
Deformation, top view
plate: beam strength
Bending in unsupported
areas peels Nevlev causing
breakage
cardboard:
bending support
Strain, side view
Add beam strength to
prevent bending
Foam rotated 90°:

cushioning support
Nevlev
Deformation, side view
Software Test Engineering
Testing Lab
Software – What is Tested?
Platform Code developed by NOSTG

Independent Common code across all hardware platforms
Code IPv4, BGP, EEM, SYSLOG, AAA, MPLS, etc.
System System
Integration Platform Code developed by Product Business Unit Performance
Test Dependent Code specific to the hardware platform Test
Code VSS, H/W Netflow, H/W QoS, H/W IPv4, etc.
Platform Functional Test
System Performance Testing
IPv4, IPv6, MPLS and Layer

2 testing
Tests latency at different

packet sizes
Tests Packets per second at

different loads and packet
sizes
Platform Functional Test
Master Test Plan sets out
overall objectives
Functional test plan tests

individual aspects
Testing Process fully

automated
Regression tests against

earlier features
Testing plan coupled closely

with test plans from software
engineering
System Integration Testing
Tries to simulate real life

operation
Do multiple features run

without impact to operation
of switch?
Combination of multiple
features running in parallel
Do You Know?
How many test scripts were

typically run for a new major line
code release for the Catalyst
6500?
Complete Your Online
Session Evaluation
 Give us your feedback and you
could win fabulous prizes.
Winners announced daily.
 Receive 20 Passport points for
each session evaluation you
complete.
 Complete your session evaluation
online now (open a browser Don’t forget to activate your
through our wireless network to Cisco Live Virtual account for access to
access our portal) or visit one of all session material, communities, and
the Internet stations throughout on-demand and live activities throughout
the year. Activate your account at the
the Convention Center.
Cisco booth in the World of Solutions or visit
www.ciscolive.com.

Brkarc 3466

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Brkarc 3466

Hochgeladen von

Copyright:

Verfügbare Formate

Engineering behind building a switch

Marketing Business Case The “Wish” List Engineering Study

Initial design concept

Foundation for ID and ASIC

Required Response Spectrum of

Mid Plane rigidity and strength is important

At the point of connection, force needs to be applied

Insertion force will bend Mid Plane

Might uncouple connector from existing installed

To Fabric Modules To Central Arbiters

Front Panel Ports

ECAD team responsible for all

Tussle with Electrical and Mech

Build and Intermediate Data

Yellow squares indicate “not to exceed”

A trace is a signal path between two components…

All trace lengths

ASIC “Routing” of traces

How many traces (paths) on

Mechanically support and electrically connect

Each layer provides a base upon which a path (net)

Vias (drill hole) exist in each layer to allow paths to

How many PCB layers make up Cisco’s new 40G

The story behind the development of

Original design from

Ground Breaking Product

Multiple Design Concepts drawn up

Initial Design Concept handed to Engineering

Engineering make changes based on thermal profile and testing

Design concepts lost in adjustments

The “Look” was turning ugly again…

Industrial Design Revision kicks in again

Revised design based on car design inspiration

New design needed to ensure it did not

Further Minor Adjustments NEW

Color Block on Bottom

FIB Table Size ECMP Paths LAG WCCP CoPP Policing

Multiple Switch ASIC’s

Processing of packets Layer 2 Processing Fabric Buffers

Ingress QoS ACL

Note : There is no one correct forwarding path!!!

Define forwarding path for each packet

Data Path and Port functions also architected

TCAM Adjacency MET SPAN VOQ’s Memories

architecture RTL of COUNTER is

2000 2003 2006 2008 2010 2011

42M 220M 291M 781M 995M 2.3B

How many Transistors in

Gate constructs assembled onto DIE space

3um Intel 8088 (1975)

1.5um Intel 80286 (1982)

800nm Pentium P5 (1989)

Intel Sandy Bridge 32nm (2011) 11nm (2016)

How many features/lines of code

Simulates drop of equipment onto a hard floor

Determine break points

Stress fractures, rivet strength

Also test drop of equipment in packaging

All equipment should function after drop

Test based on Loma Prieta Earthquake

7.2 Richter Scale

Load chassis at top of rack

Confirm function while test in progress