Campus

Design Principals
John Hicks
Internet2
jhicks@internet2.edu

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Overview
Campus design challenges
Edge Networks
Layer2 design
Core Networks
Research and EducaEon Networks
Network Management

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Campus Design Challenges

Many are not structured properly and
cant eecEvely uElize high bandwidth
REN connecEons
Many make heavy use of NAT and
rewalls that limit performance
Many are built with unmanaged
network equipment that provide no
ability for monitoring or tuning the
network

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

How Best to support R & E

Research and EducaEon needs exible and

open networks
Things to consider

NAT makes some things hard (H.323 video

conferencing)
Filtering makes it hard for researchers,
teachers, and students to do interesEng things
Your campus network must not be the
boZleneck

Make a plan for improvement without a

plan, how will you get there

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Campus Design Rules

Minimize number of network devices in
any path
Use standard soluEons for common
situaEons
Build Separate Core and Edge Networks
Provide services near the core
Separate border routers from core
Provide opportuniEes to rewall and
shape network trac

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

Core Versus Edge

Core network is the core of your

network

Needs to have reliable power and air

condiEoning
May have mulEple cores Always route in
the core

Edge is toward the edges of your

network

Provide service inside of individual

buildings to individual computers
Always switch at the edge

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Core versus Edge (cont.)

There is usually a correspondence between
building separaEon and subnet separaEon
Switching inside a building
RouEng between buildings

This will depend on the size of the network

Very small networks can get by with doing
switching between buildings
Very large networks might need to do
rouEng inside buildings

Broadening the Reach Workshop,
Kansas City, MO
02/17/14 02/19/14

Layer 2 Network Design Guidelines

Always connect hierarchically
If there are mulEple switches in a
building, use an aggregaEon switch
Locate the aggregaEon switch close to
the building entry point (e.g. ber
panel)
Locate edge switches close to users
(e.g. one per oor)

Max length for Cat5 is 100 meters

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

More Layer2 Guidelines

Minimize the distance between elements
Never daisy chain

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Start Small

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Build Incrementally

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Grow Hierarchy

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Add Redundancy

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Core Network RouEng versus

Switching Layer 2 versus Layer 3
Routers provide more isolaEon
between devices (they stop broadcasts)
RouEng is more complicated, but also
more sophisEcated and can make more
ecient use of the network,
parEcularly if there are redundancy
elements such as loops

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

Layer 3 Switches
Many vendors use the term Layer 3 Switch.
These are contradictory terms
Layer 3 = RouEng
Switch = Layer 2

What vendors mean is that it is a device that

can be congured as a router or a switch or
possibly both at the same Eme

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Core Network

Reliability is the key

Remember many users and possibly your whole

network relies on the core

May have one or more network core locaEons

Core locaEon must have reliable power
UPS baZery backup (redundant UPS as your
network evolves)
Generator
Grounding and bonding
Core locaEon must have reliable air
condiEoning

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Core Network
At the core of your network
should be routersyou must route,
not switch
Routers give isolaEon between
subnets

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Simple Core Network

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Where to put Firewalls

Security devices must be placed inline
This means that the speed of the rewall
aects access to the outside world
Try to have parts of your network non-
rewalled
This will allow full bandwidth, un-ltered
access to the Internet

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Where to put Servers?

Servers should be on a high speed
interface o of your core router
Servers should be at your core locaEon
where there is
good power
air condiEoning

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Border Router
Connects to outside world
RENs and Peering are the reason you
need them
Must get Provider Independent IP
address space to really make this work
right

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Layer 2 and 3 Summary

Route in the core

Switch at the edge
Build star networks dont daisy chain
Buy only managed switches re-
purpose your old unmanaged switches
for labs

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

No Network is Perfect
What happens to your network when
you get a 100Gbps connecEon from
Internet2?
Where are the boZlenecks?
How will you improve performance?
What is your plan?
Can you upgrade without disrupEng the
whole campus?

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

Research and EducaEon Networks

High bandwidth networks
10G, 40G and 100G backbones
Research typically needs uncongested
networks
Which means many RENs are lightly used with
lots of
unused capacity (we call it headroom)
Low latency
Terrestrial ber
Open Networks with no ltering
Firewalls can make it hard for ad-hoc acEviEes

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

Why a REN?
Enable research or services that could

not be accomplished otherwise

Cost Savings (buyers club)
Aggregate demand from mulEple
parEes
Vision of building alliances
Successful RENs nd that there are
unanEcipated benets

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

Why Are We Doing This?

Our goal is to build networking capacity to
support Research and EducaEon
Remember: University = Research & EducaEon

Buying all service from your local ISP is a losing

game
you will spend more money and not have control of
the network

The paZern around the world is to build

regional, naEonal, and larger Research and
EducaEon Networks (RENs)

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

REN versus Campus Network

The Campus Network is the foundaEon
for all Research and EducaEon acEvity
Without a good campus network, the
Research and EducaEon Network cant
work as well as it should
The campus network is the foundaEon
that the REN is built upon

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

REN Ecosystem

A layered model
Global ConnecEvity
Regional RENs
NaEonal Research and EducaEon
Networks
All users are connected at the campus
network level
No scienEst is connected directly to
a NaEonal Network. They are all
connected to campus or enterprise
networks

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Network Management
Keeping Track Of StaEsEcs
For purposes of accounEng and metering

Faults (Intrusion DetecEon)

DetecEon of issues,
TroubleshooEng issues and tracking their history

TickeEng systems are good at this

Help Desks are a useful to criEcal component

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

ExpectaEons
A network in operaEon needs to be monitored in
order to:
Deliver projected SLAs (Service Level
Agreements)
SLAs depend on policy
What does your management expect?
What do your users expect?
What do your customers expect?
What does the rest of the Internet expect?
Whats good enough? 99.999% UpEme?
There's no such thing as 100% upEme

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

Baselining
What is normal for your network?
If youve never measured or monitored your
network you will need to know things like:
Typical load on links
Level of jiZer between endpoints

Typical percent usage of resources

Typical amounts of noise:
Network scans
Dropped data
Reported errors or failures

Broadening the Reach Workshop, Kansas City, MO
02/17/14 02/19/14

Why do all this?

Know when to upgrade

Is your bandwidth usage too high?

Where is your trac going?
Do you need to get a faster line, or more providers? - Is
the equipment too old?

Keep an audit trace of changes

Record all changes

Makes it easier to nd cause of problems due to

upgrades and conguraEon changes

Keep a history of your network operaEons

Using a Ecket system lets you keep a history of events.

Allows you to defend yourself and verify what
happened

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Why manage Networks?

AccounEng
Track usage of resources
Bill customers according to usage
Know when you have problems
Stay ahead of your users! Makes you look good.
Monitoring soqware can generate Eckets and
auto-maEcally noEfy sta of issues.
Trends
All of this informaEon can be used to view trends
across your network.
This is part of baselining, capacity planning and
aZack detecEon.

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

Special thanks to
NSCR for slide material and organizaEon
GRNOC, Indiana University

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14

QuesEons?

Broadening the Reach Workshop, Kansas City, MO

02/17/14 02/19/14