Seven Pillars

of CarrierGrade Security
in the AT&T Global IP/MPLS Network
INTRODUCTION
AT&Ts legacy and expertise lies in the creation and maintenance of secure, reliable
networks that are always on and available when you need them. This goal is as valid today
for our Internet Protocol (IP) and MPLS (MultiProtocol Label Switching) networks as it was
for traditional circuit switching network. And now that IP /MPLS networks are embedded in
the critical processing of our business and government customers applications, we are
more committed than ever to ensuring superior levels of carriergrade security for our
customers especially those using IP.
More specifically, in support of AT&Ts Global IP/MPLS network, AT&T s Chief Security
Office has developed a set of seven basic security protection methods or pillars, as we
refer to them. These firm pillars maintain a constant security focus in all of our design,
deployment, and operational processes around our IP/MPLS core.
This brief article explains these security building blocks and principles that are inherent in
our IP/MPLS backbone network. In particular, we provide an introduction to our security
methodology rooted in what we call the Seven Pillars of CarrierGrade Security in the AT&T
IP/MPLS Network: Separation, Automation, Monitoring, Control, Testing, Response, and
Innovation.

Pillar 1: Separation
Customer traffic is separated using MPLS and the concept of Virtual Private
Networks
AT&Ts MPLSbased VPN services AT&T s Virtual Private Network (AVPN) service and its
legacy MPLS VPN products, IPEnabled Frame Relay Services, Enhanced VPN Service
(eVPN), and MPLS Private Network Transport (PNT) are all designed to take advantage of
the inherent separation strength of MPLS and MultiProtocol BGP (MPBGP)

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

AT&T's standardsbased implementation assures that data packets and customer specific
routing information cannot leak" out of, or into, one customers VPN to another. Or, from
a customer's VPN to AT&T control traffic on our backbone.
What this means is that customers need not be concerned with unauthorized disclosure or
modification of their sensitive VPN traffic (or routing information) by other IP/MPLS
network users. Several MPLS standards have been specifically published to prevent any
type of unauthorized or illegitimate crossVPN sharing of customer routing or data traffic.
Independent tests performed by many companies including Cisco, Miercom and AT&T
repeatedly have confirmed this important security requirement.1 Sadly, many backbone
network providers suggest that by simply not allowing Internet peering links to directly
attach to their private IP networks, they can mix together customer IP traffic using
different less secure methods. This naive approach conveniently ignores the fact that the
major source of many Intranet security violations to date has come not from direct attacks
on a carrier's backbone infrastructure, but rather from customer networks and/or network
management systems that attach via "backdoor" connections to the Internet. Eightfour
percent of enterprises and government agencies reported some type of security breaches
in the last year (2009), according to a new survey by Computer Associates International.
The survey also found that security breaches have increased 17 percent in the last three
years, according to the survey released by the Islandia, N.Y., IT management software
company on July 5.
Customers of AT&Ts MPLSbased VPN services can expect that their own VPN network will
have the following basic security characteristics derived directly from the strengths of
MPLS and MPLS VPN related standards (e.g., RFC 4364):
-

Containment: Traffic (and routing information) sent between customeredge (CE)

routers on the same VPN will always stays within that specific VPN no spillover or
"leakage" can occur.
Isolation: No customers VPN can in any way materially affect or influence the
content or privacy of another customers VPN
Availability: Aside from the basic security related attributes of MPLS and MPLS VPNs,
AT&T carefully engineers shared resources to meet the highest levels of availability
and mitigates potential denial of service activities through additional methods such
as access control lists, route filters, turning off unnecessary services, and other
infrastructure hardening techniques.

Note: MPLS security testing referenced here was performed in the early 2000s; AT&T and Cisco
testing are proprietary. The Miercom paper is found here: http://www.noida.stpi.in/mercommpls.pdf
2
http://www.eweek.com/c/a/Security/Study-Security-Breaches-Afflict-Most-Enterprises-Governments/
2011 AT&T Intellectual Property. All rights reserved.
AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

Simplicity: MPLS networks allow for simplified provisioning in both the customer and
carrier domains (and hence can help to avoids security related configuration
mistakes). First, MPLS VPNs are much simpler for customers to configure than legacy
Layer1 (e.g., private line), Layer2 (e.g., Frame Relay or ATM) pointtopoint
solutions, or Layer3 (e.g., IPSec VPNs). Second, MPLS VPNs allow for much more
scalable service provider architectures, unlike some other VPN solutions (e.g., L2TP)
based on ACLs and separating customer address space. A service provider network
using access control lists or separate IP spaces as the primary method to create VPN
separation has a very difficult task to manage. In this scenario, every new site or
route that is added can potentially require a change on every other router in the
network to ensure security. This is not a scalable solution and can lead to errors in
configuration and potential security breaches. In summary, the most scalable MPLS
based architectures allow the service provider to more cost effectively provide
reliable, highperforming services to a large number of enterprise customers
without disrupting their existing customer base as organic growth occurs on the
providers network.

The diagram below illustrates the AT&T Global MPLS Network Architecture.

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

This architecture incorporates several key components to ensure the security and reliability
of the VPN customers.
Security/Privacy
In order to provide the highest level of security, separate "edge" routers are used in the
AT&T network for VPN customers versus public Internet customers. The VPN edge routers
are physically separate devices that only support private MPLS VPN customer connections.
AT&T's MPLS VPN services are based on RFC 4364 (formerly RFC 2547) and provides privacy
equivalent to Frame Relay or ATM Service according to most industry experts.
AT&T's standards based MPLS network architecture provides:

VPN route uniqueness and segregation using MPBGP attributes such as: Route
Targets (RTs) which are used to control route distribution of customer specific
VPN routes into their own dedicated route table or VRF. Customers are
unaware of the RDs and RTs and VRFs associated with their VPN
Route Distinguishers (RDs) which are appended to a customer's routes to help
identify them as belonging to a specific VPN.
Virtual Routing & Forwarding Tables (VRF) Each customer's VPN routes are stored
in a separate and unique routing table.
VPN membership and all network configurations are controlled by automated AT&T
provisioning systems.
Customer specific interfaces on each PE are automatically assigned to that particular
customers VPN. Several consistency checks are made in systems to ensure that any
new connection or VPN is legitimate for that customer.

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

Reliability
While the edges for VPN and Internet services are physically separate, the AT&T IP/MPLS
core is shared across these services. To ensure security and reliability several architectural
design steps have been taken:
1.RouteFree Core: The core network only provides Label Transport over Label
Switched Paths. The routers have no knowledge of any Internet or customer specific
VPN routes and IP addresses.
2. Control Plane: No backbone routers are visible to the Internet or customer
specific VPN and none can be reached from any external endpoint.
3. Data Plane: VPN and Internet traffic is carried across separate ELSPs so that
different traffic or service types can be differentiated in the core. For example, if a
large internet DOS type of event were to occur, VPN capacity is protected to ensure
no adverse impact to a customer's private VPN traffic.

Pillar 2: Automation
Automated perimeter security tools protect AT&Ts MPLS core
Over the past several decades, AT&T has invented, patented, and perfected many
automated tools and systems to manage and protect its telecommunications infrastructure
and networks. AT&T has continued to invest in automated methods for provisioning and
maintaining its global IP/MPLS network and services. This included the development of
several patented IP and MPLS troubleshooting tools, and further automation surrounding
detection of security anomalies. Seamless integration and automation has been one of our
key strengths. AT&T also knows, however, that in the creation of any new network
infrastructure or service, that some manual techniques must precede such automation.
That is, before such critical tasks such as automated provisioning or change management
can be integrated into a carrier backbone environment, manual processes must first be
documented, tested and perfected. Then, and only then are the systems and tools
developed to carry forward these proven methods and best practices.
One of AT&Ts key advantages in the inevitable convergence of global network
telecommunications traffic to MPLS is that we have made significant progress in moving
through the experience curve toward the use of automation. Stated more simply: We are
way ahead of our competitors in the charge toward flawless use of automated
management techniques.

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

AT&T has had many years of experience with MPLS, and was an early adopter, announcing
its first MPLSbased service in 1999. Since then, AT&T has continually rolled out new and
enhanced MPLSbased IP VPN services in support of enterprise customers. Today, AT&T is
regarded by leading telecommunications analysts as having one of the most comprehensive
VPN portfolios in the industry, including MPLS, IPSec and SSLbased solutions.
AT&T has put in place a number of specific security measures, supported by automation, to
support its global IP/MPLS network infrastructure. These include:

Filtering: AT&T uses standardized engineering rules and automated provisioning

systems to manage infrastructure specific access control lists (ACLs) related to
access to the network as well as control traffic across the network.
Least Privilege: Infrastructure routers, and PE interfaces, are hardened by turning
off, or severely restricting, unnecessary protocols and ports.
BGP Authentication: Border Gateway Protocol (BGP) authentication can be
implemented on upon customer request for many services CEPE eBGP sessions.
BGP authentication ensures that the BGP routes passed between two BGP speakers
(routers running the BGP protocol) are authorized and have not been tampered with
in any way
Limits: per session and per VPN routing prefix limits, dampening, and other
mechanisms are used on many services to limit either the rate, or total number of
routing update transactions that can be processed by an AT&T edge router.
Authentication: TACACS+, tokens, SSH and other mechanisms are used to control
access for authorized AT&T employees to access infrastructure devices.

Pillar 3: Monitoring
IP traffic monitoring provides early warning of Internet worms, botnets,
and denial-of-service.
One critical component in the management of largescale network traffic is the generation
and analysis of traffic flow data to detect trends and anomalies. Such exceptionbased
processing has become the basis for many new forms of intrusion detection. AT&T has
been using this technique for a number of years to identify patterns of normal network
behavior and to measure differences from observed patterns. AT&T has the most
extensive commercially available infrastructure in the industry for detection of traffic
anomalies that are indicative of denial of service attacks. As well, AT&T uses proprietary
2011 AT&T Intellectual Property. All rights reserved.
AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

technology to detect patterns that indicate worms, botnet command & control, and other
anomalies.
Since the late 1990s, AT&T has used these technologies to identify clear network patterns
of anomalous behavior leading up to the Slammer, Blaster, Nachi, and SoBig worms and
viruses. This was accomplished through a proactive, 24/7 analysis of network flow data
(data content is not necessary for such profilebased security). In some cases, clearly
recognizable spikes occurred days before large events. AT&T has continued to develop this
technology that has helped detect alert customers promptly of changes in
Conficker/Downadup worm behavior in late 2008 and into 2009. Detection of malicious
botnet command and control as the botnets recruit new zombies allows AT&T to take a
more preemptive approach to network security and attack prevention.
This monitoring provides unique protection benefits for the MPLS network in two ways.
First, it allows our security teams to take steps toward the appropriate filtering often well
in advance of other providers. And second, by tasking the monitoring system to detect any
probes aimed at the MPLS core address space, weve invented a novel means for
dramatically reducing risk in our core. Customers of AT&Ts MPLSbased services thus can
enjoy the following benefits of our monitoring systems:
Anomaly Detection: AT&T proactively monitors traffic for anomalies that provide
evidence of worm and virus trends in real time.
External Access: AT&T also alarms and monitors infrastructure elements for
resource consumption and attacks.
Analysis: The worldclass statisticians from AT&T Laboratories Research continue
to make great strides in algorithms for security anomaly detection.
By virtue of the MPLS network, AT&T has developed a suite of optional security services
that complement and can be used in conjunction with MPLSbased VPN services. Private
Intranet Protect service is available to provide flowdata analysis of an enterprises own
VPN environment. The service does not require any additional equipment to be deployed
at your sites, which keeps costs down and reliability high. AT&T can provision systems in
the core network to gather flow record data (no content), pass the flow data to analysis
systems, and provide alerting and analysis through a web interface that is specific to an
enterprises network traffic. Networkbased Firewall is another service that can be used to
safely access the Internet directly from an enterprises MPLS VPN with packet inspection
and IDS with options for, user authentication, IPS, web content filtering, email scanning,
multiple DMZs hosting, and DDoS Defense services. All of these services are facilitated
through MPLS capabilities. Security Network and Operations Center (S/NOC, SOC)
functions as well as premisebased solutions can also be provided to provide a full
2011 AT&T Intellectual Property. All rights reserved.
AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

complement of security enforcement and monitoring capabilities throughout an

enterprises network. Advanced security analysis and threat management is available
through AT&T Security Event & Threat Analysis (SETA) providing prioritized alerting based
on correlated analysis of logs and alerting from multiple network devices, device types, and
applications.

Pillar 4: Control
AT&T enforces strict operational security controls in its MPLS core.
24/7 network operations strike at the heart of the basic value proposition AT&T offers its
customers. Weve been in the business for over a century and while technologies and
customer needs have changed dramatically through these years, one thing has remained
constant in our service provision: Operational Excellence.
Our original experiences moving massive volumes of customer data over our first high
speed packet networks for Frame Relay and Asynchronous Transfer Mode (ATM) networks
exemplifies this operations focus. When these technologies became popular in the early
nineties, one common criticism was that the reliability associated with these networks
could never approach the excellence achieved in circuit switched environments.
The good news is that after our initial period of growth (and yes there were occasional
operational errors during this period), we managed to achieve levels of reliability and
availability in these networks that exceed that of our circuit switched networks. Our MPLS
core is no different. When we chose to build our first IP network and then enable it with
MPLS technology, we applied the same relentless focus on reliability and resiliency to that
network, both in the U.S. and globally. Today our IP/MPLS network consistently achieves
higher reliability than traditional TDM voice, which used to be considered the gold standard
for reliability. We have achieved this high reliability in a manner that is simpler to provision,
easier to operate, and more difficult to attack than any network technology weve ever
operated. So the basic elements of our operational excellence in support of MPLS security
are as follows:
Administrative Separation All MPLS network management traffic is isolated on a
separate MPLS VPN, using loopback addresses provisioned on a separate MPLS VPN,
from protected address space which is not advertised; all traffic to this protected
address space is blocked at the edges of the network and is not visible from the rest
of the network.

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

Processes: AT&Ts operations follow mature Methods and Procedures (M&Ps) that
are derived from decades of best practices in operating carrier networks.
Root Cause Analysis: All incidents are subject to comprehensive Root Cause Analysis
steps a process used to ensure process improvements through any operational
policy violations.

Pillar 5: Testing
AT&T uses testing, audits, and reviews to ensure security compliance.
Our Information Security team employs some of the best ethical hackers on the planet.
These engineers are tasked with the constant chore of probing, testing, and trying to find
weaknesses in our MPLS network. Occasionally they find an area in which improvements
are necessary and steps are taken immediately to address their findings.
In addition, our Information Security team works with both internal and external auditors to
ensure that all operations and infrastructure teams follow the industrys best security
requirements. This is an ongoing task that sweeps through all aspects of our infrastructure
including the MPLS network. The mature AT&T Security Policy Requirements (ASPR)3
stands as the basic guide for all these activities.
Our processes also include the use of expert reviews and organizational approvals as so
called Security Gates in almost everything we do. Our design and development efforts, for
instance, follow a corporatewide standard and documented methodology. The ASPR
process mandates an expert security review such that newly developed processes will not
even pass the first conceptualization step without approval from designated teams of
security experts. The result of this discipline is the following for our MPLS customers:
-

Testing: AT&T conducts ongoing intrusion detection, audits and penetration testing
against server complexes for network management, customer care and service
support. Customer MPLS VPNs are created and configured by an automated
provisioning system, and any changes or discrepancies in router configuration, from
that in the backend provisioning database, will be detected by regular discords
detection/reports.
Auditing: Ongoing independent audits by independent, internal security teams are
used to confirm compliance with the AT&T Security Policy Requirements.
Reviews: All processes have embedded controls that require expert security reviews.

AT&T Security Policy and Requirements establish the security controls necessary to protect
computing and networking environments across all AT&T working environments.
2011 AT&T Intellectual Property. All rights reserved.
AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

Pillar 6: Response
AT&T deploys proactive response teams trained in the details of MPLS.
Security incident response at AT&T is performed using a tiered operations structure. AT&T
utilizes a mature, global threetiered 24/7 security operations team that is centrally
coordinated in the Global Network Operations Center in Bedminster, NJ. Expert Tier 3
security analysts support this structure as incidents are escalated using welldefined
security methods and procedures. At Tier 1, trained operations managers use mature
monitoring tools to proactively identify conditions that might warrant response. A Tier 2
management interface oversees this activity and is used to tie together conditions that
might be brewing in disparate locations.
When the appropriate condition has been identified perhaps a spike or anomaly in traffic
Tier 3 activity is initiated. From a security perspective, Tier 3 analysts are among the most
senior and trained security engineers in the company. They make the real time
determination as to whether our AT&T Computer Security Incident Response Team (CSIRT)
must be initiated. The AT&T CSIRT is a mature 24/7 operational structures and set of
processes in which experts from the AT&T Network Security team coordinate real time
response activities with operations staff from the various parts of AT&T's business. The
ACSIRT centrally manages and coordinates all response activities related to proactive
mediation based on early indicators, as well as mitigation of any detected security problems
Were proud that the current state of the practice for our ACSIRT involves proactive
response to conditions often long before customer impacts can ever become visible. In
fact, for several years AT&T has extended this proactive response to our customers
through a service called AT&T Internet Protect. Thus, customers who subscribe to this
service can rely on incident response protection from AT&T as follows:
-

Tiered Response: Incidents are dealt with via a mature tiered response infrastructure
that includes senior security and operations experts.
Proactive Indicators: The AT&T Computer Security Incident Response Team acts
routinely in a proactive manner on indicators that typically precede any customer
visible problems.
Innovative Customer Notification Service: AT&T has extended this capability to
customers through an innovative notification service that provides real time
indicators of anomalous behavior or detected security incidents to clients on a 24/7
basis.
WAN Analysis: Private Intranet Protect (PiP) option allows, with permission, AT&T

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

the ability to perform analysis across customer MPLS VPNs and notify administrators
when potentially harmful traffic patterns are detected.

Pillar 7: Innovation
AT&T funds the most extensive MPLS security research in the world.
AT&Ts research laboratory has evolved over the past two decades from a Bell Laboratories
that was involved in a broad range of technologies to a more focused AT&T Laboratories.
One of the most important issues of concern for this organization continues to be network
operations. In fact, AT&Ts laboratory is the only organization of its kind rooted in the
excellence and tradition of Bell Laboratories, but guided by the daytoday needs of our
operations teams.
And security is one of our research laboratorys key focus areas. MPLS, in particular,
provides a landscape on which our researchers have tried to find new techniques for
protecting our customer traffic and systems. Creative means for analyzing anomalies,
algorithms for integrating control and data plane information, and new means for MPLS
management and monitoring are among the many areas in which our researchers are
actively working, publishing, and sharing in the community.
As such, this research laboratory complements our development, engineering, and
operations teams in a way that remains unique in our industry. No other service provider on
the globe maintains the type of research commitment to networking and more specifically
to MPLS than AT&T. This serves to underscore our commitment to excellence in this area
and will ensure that the best available innovations are always embedded into the MPLS
infrastructure we use to support customers.

Conclusion
Our Seven Pillars of CarrierGrade Security for MPLS result in a set of conclusions that we
view as critical to our value proposition for our customers:
-

Security Equivalence: We are proud to report that AT&Ts MPLS security is currently
equivalent to the type of security provided on other technologies such as layer 2
services. This does not mean things are perfect but it does point to great advances
made in the past few years.
Continued Improvement: AT&T has always dedicated itself to programs of continued
improvements to security and will continue coming down the experience curve for

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

MPLS just as we did for Circuit Switched, Frame Relay, ATM, and Managed IP
Networks.
Appendix A includes some common security/reliability questions and the mitigation
processes.

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

APPENDIX A

Common Reliability Security Concerns

Concern
Privacy/Intrusion:
Can someone break into my VPN?

Denial of Service:
Can an Internet Traffic Storm, such as a DDOS
attack affect my network or performance?

Mitigation
1. MPLS VPN endpoints are provisioned with same
privacy level as Frame Relay/ATM.
2.VPN Edges are separate from Internet Edges
3.No Customer routes are visible on the backbone,
only labels on the backbone
4.Rigorous Security Procedures in place for
Provisioning and Maintenance
VPNs and router configurations are
autoprovisioned avoiding human touch error.
Operational Support Systems Firewalled
All access/changes are logged
Automated Discord Checks
Established incident response procedures
Documented
Deploying active intrusion detection
1.VPN Edges are separate from Internet Edges
2. Backbone is segmented so Internet and VPN
traffic are in separate ELSPs. An Internet storm
only impacts the Internet.
3.DDOS detection and proactive filtering even on
Internet

Intrusion/Reliability:
Can bogus routes bring down an edge?

1.VPN Edges are separate from Internet Edges

2.Each VPN is route limited (<20K routes)
3.Route filtering is done on Internet endpoints
and peering points to filter out bogus routes

Core Protection:
Can the backbone network be compromised?

1. Routing elements in the core are not visible or

reachable. Backbone is Internet and VPN route
free.
2.ACLs used at edges to protect core elements
3.Route filtering (Antispoofing) applied on all
inbound access points
4. Limited management access to specific ports
(Telnet, SNMP), all others turned off. Require
encrypted authentication and auditing.
5. Management access to equipment uses a
separate network.
6. Internal servers are firewall protected.

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property

Core Protection:
Can a POP be compromised?

1.Guarded and hardened AT&T facilities

2.Access requires authentication through
advanced security systems
3.All hardware is redundant
4.All POPs equipped with dual commercial power
supplies with generator and battery backup
5. Protected by AT&Ts Network Disaster Recovery
Program

Monitoring:
Is Security Monitored?

1.Elements monitored 7x24x365 by multiple NOCs

2.On site vendor support
3.All access monitored and logged
4.Customer alerts and notification

Increasing Privacy:
Can I increase my security/privacy level?

1. Customers can optionally add encryption

services (e.g., IPSECbased) to compliment the
MPLS VPN to further increase privacy out to the
edge.
2.Enhanced Security Services offered:
Firewall Services (Network and CPE based)
Intrusion Detection
Network Scanning services
Private Intranet Protect
Authentication and Directory Services
Security Event & Threat Analysis
Professional Services

2011 AT&T Intellectual Property. All rights reserved.

AT&T and the AT&T logo are trademarks of AT&T Intellectual Property