Beruflich Dokumente
Kultur Dokumente
FreeBSD® and the FreeBSD Logo are registered trademarks of The FreeBSD Foundation.
IBM® is a trademarks or registered trademarks of International Business Machines Corporation in the United States and/or other
countries.
Juniper Networks® and Junos® are [registered] trademarks of Juniper Networks, Inc. in the United States and other countries.
Windows® is either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
Facebook Network Appliance Deployment | 2
Facebook Network Appliance Deployment | 3
Facebook Network Appliance Deployment | 4
Thank you for choosing to install the Facebook Network Appliance (FNA)!
Facebook Network Appliance is Facebook’s content caching program. FNA provides Internet Service Providers
(ISPs) with an efficient means of delivering static Facebook content from within their existing network. Upon
deployment, an ISP will offload a significant amount of Facebook content from its backbone network and vastly
improve the Facebook user experience.
The FNA kit consists of a Cisco Nexus Top-of-Rack (ToR) switch, four Hewlett Packard ProLiant DL380
servers, and their required interconnects. The hardware is suitable for deployment in data centers, colocation
facilities, and outside plant environments (the industry standard 19-inch form factor allows integration into most
network environments).
This document describes the procedures necessary to install, configure, and operate the FNA.
Facebook Network Appliance Deployment | 5
This section lists equipment required for FNA installation. Ensure that you have received all of the equipment
required for either your SKU1 or SKU2 order (Tables 1 and 2, respectively).
The following table lists equipment that is included with a SKU1 order. If equipment is missing, refer to the
Support section for contact information and seek assistance.
The following table shows equipment that is included a SKU2 order. If equipment is missing, refer to the
Support section for contact information and seek assistance.
NOTE: SKU2 does not include a Cisco Nexus switch. However, the order includes SFP+ optical transceivers for
additional uplink capacity. Use the existing switch at your FNA deployment.
NOTE: The quantity of servers you receive depends on your current and forecasted Facebook traffic volume.
Servers are shipped in groups of four. The equipment will arrive at your facility in one shipment. IBM (FNA’s
logistics partner) will contact you with shipping information.
Facebook Network Appliance Deployment | 7
This section shows additional equipment required for the FNA installation that is not included with a SKU order.
Table 3: FNA SKU1 and SKU2 Equipment Required but Not Included
(2) Fiber Patch Cables from Cisco Nexus 3132Q-X switch to router1,2
1The FNA accepts both SR and LR optical transceivers. Facebook provides the most optimal
optical transceiver type corresponding to the type you currently use to connect your routers.
2
Fiber connectivity on the FNA side is the Lucent Connector (LC) type. Fiber connectivity on the
router side can be LC-LC, LC-SC, (Lucent Connector – Subscriber Connector), etc.
Facebook Network Appliance Deployment | 8
This section describes how to install equipment to build an FNA cluster. After installing the equipment, you will
need to configure the equipment to work in your facility. This information is provided in section 4 Network
Configuration.
Use the pre-installation checklist to confirm that your environment has the resources required to install the FNA
into your existing network:
Using the supplied network cables, connect each ProLiant DL380 server from network interface
controller (NIC) port 2 to their respective QSFP+ ports on the Cisco Nexus switch (see and 3).
Begin with the server at the highest location in the rack (labeled fna001) and work your way to
the bottom-most server. Connect fna001 to port 32 on the switch, fna002 to port 31, and so on.
See Figure 30 in Appendix F: Additional Installation Images for a reference on server
connectivity order.
NOTE: Each ProLiant DL380 server and QSFP+ connector have a round green sticker to show the correct
orientation. Ensure both green dots on each QSFP+ connector and server chassis are facing upward.
NOTE: The Cisco Nexus switch QSFP+ ports 9 ~ 12 are not used and should not be utilized for any FNA or
non-FNA equipment. Each Cisco Nexus switch supports up to twenty servers.
Facebook Network Appliance Deployment | 10
NOTE: The Cisco Nexus switch does not have power buttons. They receive power when connected to a PDU with
adequate power.
Facebook Network Appliance Deployment | 12
Check and record the LED states for each PSU, NIC, and Solid State Drive (SSD) for each
ProLiant DL380 server in the cluster. For LED definitions, refer to the LED Definitions section in
Appendix B: Troubleshooting.
Check the light levels (for deployments utilizing a Cisco Nexus switch, refer to Measuring the Optical
Level through IOS, Document ID: 16150). Light levels need to be between -2dB and -7dB for proper
operation. Light levels outside of this range may be indicative of a connectivity issue. If you find the
levels out of range, see Appendix B: Troubleshooting.
This completes the FNA hardware installation procedure. Equipment installation should look similar
to Figure 7.
Facebook Network Appliance Deployment | 13
NOTE: If you are adding additional equipment to your initial FNA installation, see section 6 System Upgrades.
Facebook Network Appliance Deployment | 14
This section describes the procedure to configure the FNA to peer with Facebook’s network.
Connect a VGA monitor and USB keyboard to one of the FNA servers. Upon server boot-up, you will
see the FNA Main Screen in Figure 8:
NOTE: The servers should be preconfigured with IP addresses. If this is not the case, you will need to create and use
a custom USB boot disk. See Section 9.1 Creating a Using a USB Boot Disk.
Configure the point-to-point connection. This connection will be configured on the link-aggregation
control interface (LACP). See Appendix C: Link Aggregation/Link Aggregation Control Protocol
(LACP) Configuration for additional configuration details.
Untag the Virtual LAN (VLAN) link-aggregation interface.
Set a static route to the allocated subnets via the link-aggregation interface.
NOTE: Facebook requires that the FNA have internet connectivity before BGP peering can be established. The
BGP peering session does not need to be routing before the FNA has internet connectivity.
Configure BGP.
Use the IPv4 and IPv6 addresses originally supplied in the FNA order form.
Enable Graceful Restart (GR). See Appendix D: BGP Configuration Details for configuration
information.
Enable External BGP (EBGP) Multi-hop.
The Autonomous System Number (ASN) that you will be peering with is 63293.
Configure your router’s BGP settings. See Appendix D: BGP Configuration Details for
configuration information.
Facebook Network Appliance Deployment | 15
NOTE: Once the router’s BGP settings have been properly configured, a latency based routing test will be initiated to
optimize routing paths. It is required that all FNA servable netblocks be announced for the purpose of this test. This
process takes about twenty-four hours to complete. BGP establishment is not required during this time. See
section 4.4 Latency Based Routing for more information.
NOTE: The BGP peering session is intended to be a router-to-server configuration and not a switch-to-switch
configuration.
FNA applies inbound filtering to reject the following address spaces on BGP peering sessions to ensure
successful BGP establishment:
Default route
Bogon prefixes
RFC 1918
Facebook’s IP prefixes
Facebook Network Appliance Deployment | 16
The FNA utilizes a latency based routing test to optimize traffic routing. For the purpose of this test, it is critical
that all netblocks that can be served be announced to the FNA. Note that prefixes announced to the FNA
only implies those prefixes are eligible to serve Facebook traffic.
The latency based routing test evaluates the round-trip latency of your network. This is accomplished by
sending a small sample of static content from the netblock population you announced. Initially, only a small
sample set is sent to the FNA cluster. The sample volume will ramp up during this test. This makes it possible
for Facebook to observe and construct the most optimal routing paths for your FNA deployment.
The accuracy of the latency based routing service is dependent on the percentage of FNA servable netblocks
announced. Providing a large percentage of your FNA servable netblock population allows the service to
construct more optimal (low-latency) routing paths. A small percentage of netblocks announced (i.e., a test
netblock) will not allow the test to function as intended and traffic may be routed in a seemingly random course.
It is critical that every FNA servable netblock is provided to Facebook to optimize routing paths for
your FNA deployment.
Facebook Network Appliance Deployment | 17
Please complete this checklist after you have installed the FNA to verify proper installation.
This section describes how to add additional capacity your original FNA deployment.
WARNING: This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause
service interruption. If this is the case, it is recommended to drain network traffic first. See section 7 Maintenance and
Monitoring for instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the
hardware is available.
Rack the additional ProLiant DL380 servers, connect the peripheral cables (power and network), and
power on the servers. Utilize section 3 Hardware Installation for reference.
Connect (2) additional 10G fiber uplink cables from port 3 and port 4 on the Cisco Nexus switch to the
same router (or switch) where the initial two 10G fiber uplinks are connected (see Figure 9).
Update the Link Aggregation (LAG) configuration on the router (or switch) to accommodate the
additional interfaces.
NOTE: All cluster augments are implemented with an increment of four ProLiant DL380 servers.
Facebook Network Appliance Deployment | 19
WARNING: This procedure requires updating the Link Aggregation (LAG) configuration. Updating the LAG may cause
service interruption. If this is the case, it is recommended to drain network traffic first. See the Maintenance section for
instructions on how to drain network traffic.
Refer to section 2.2 SKU2 Provided Equipment for a list of provided hardware. Make sure all of the
hardware is available.
Rack the ProLiant DL380 servers, connect the peripheral cables (power and network), and power on
the additional servers. Utilize the Hardware Installation section for reference.
Connect the provided QSFP+ to 4x SFP+ breakout cable assembly to port 2 on the Cisco Nexus
switch. See Figure 10 for breakout cable interface location.1
If you are upgrading from a (16) server to a (20) server cluster, connect the 2nd provided
QSFP to 4x SFP+ breakout cable to port 3 on the Cisco Nexus switch.
Connect (2) additional 10G fiber uplink cables for each group of (4) servers from the Cisco Nexus
switch to the same router (or switch) where the initial 2x 10G fiber uplinks are connected.
Update the LAG configuration on the router (or switch) to accommodate the additional interfaces.
1
NOTE: The provided QSFP to 4x SFP+ breakout cable assembly only supports LR single-mode connections.
NOTE: Refer to Figure 32: System Architecture Diagram for a diagram of a (20) server FNA cluster.
Facebook Network Appliance Deployment | 20
This section provides a reference for best practices regarding augmentations. This growth path optimizes
for deployment failover scenarios by reducing the quantity of Single Points Of Failure (SPOF).
NOTE: When building an FNA cluster beyond eight servers, it is best practice to install the additional servers in a new
rack and maintain equal size clusters (as depicted in growth path 3 and 4).
Facebook Network Appliance Deployment | 21
When performing either singular or recurring maintenance jobs, email fna-ops@fb.com to report the planned
maintenance.
Before powering down the FNA cluster for maintenance, all network traffic must be properly drained. This
procedure describes how to properly drain traffic from the cluster:
IMPORTANT: This procedure requires withdrawing prefixes. It is important that the BGP peering session is
maintained while withdrawing prefixes.
Withdraw prefixes.
While maintaining the BGP peering session, begin withdrawing BGP prefixes.
Traffic will begin to drain.
Traffic will fully drain in less than one hour.
You should shut down the FNA only if it is absolutely required and traffic has been properly drained from the
system (see 7.2 Draining Network Traffic). This should be a rare occurrence. To shut down the FNA, follow
the procedure outlined below or contact fna-ops@fb.com for assistance:
Connect a monitor and keyboard to the server you want to power off.
Use the arrow keys navigate to the Shut Down text.
Press the [Enter] key. The FNA will power off.
Facebook regularly monitors internet reachability from various simulated endpoints. When reachability issues
are detected on these Virtual IPs (VIPs), Facebook may drain traffic from an FNA cluster. This measure
ensures that Facebook maintains a high quality of service for the Facebook platform.
Facebook Network Appliance Deployment | 22
The Terms of Service agreements that you signed when you installed the FNA in your facility included specific
language pertaining to the disposal of storage media. That language is replicated here for your convenience.
Only deal with and process Edge user data in compliance with, and subject to, the instructions received
from Edge and will not use or process the Edge User Data for any other purpose whatsoever.
Take custody of and secure, and support the destruction of, all server storage media (example: hard
disk, solid state disks, flash memory cards, etc.) that are uninstalled from the servers, and provide
secure handling, storage, packing, and shipping of the storage media per IBM provided instructions and
at IBM's expense to an IBM designated asset recovery center for destruction.
Facebook Network Appliance Deployment | 23
FNA equipment comes preconfigured with the base software installation and IP configuration that was
specified in the FNA kit order. In some instances, the IP configuration may need to be changed (such as
configuration changes made between processing an order and receiving the kit). If this is the case, you will
need to reconfigure the equipment by creating and using a USB Boot Disk.
To create a USB boot disk with to change the system configuration, follow these steps:
Locate the name of the FNA kit that needs to be reconfigured under the Installed Kits pane. Verify that
the FNA IP Prefix and every P2P address are correct.
If any information is incorrect, stop and inform the FNA team of the changes that need to be
made (see section 11 Support). Only continue to the next step after the information has
been corrected.
Locate the USB Boot Disk item under the Installed Kits pane. Click the Disk Image and Kit Config
links to begin downloading (see Figure 13).
Facebook Network Appliance Deployment | 24
Connect a blank USB flash drive into a computer running either Windows, MacOS, or Linux.
Instructions for creating a USB boot disk for each of these operating systems is provided in this section.
Create the USB boot disk. Utilize the instructions for the appropriate operating system:
WARNING: This drive will be completely overwritten.
Use Win32 Disk Imager to write the Disk Image to the appropriate USB drive.
Run diskutil list to find the mount point of the USB drive.
Diskutil list
/dev/disk0 (internal, physical):
/dev/disk1 (internal, virtual):
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: FDisk_partition_scheme *15.5 GB disk2
1: DOS_FAT_32 UNTITLED 15.5 GB disk2s1
Unlocked Encrypted
Confirm you have selected the correct mounting point by running diskutil info disk# and
verifying the drive name, size, and protocol.
Run diskutil unmountDisk <disk#> to unmount the USB drive.
WARNING: dd will overwrite any disk you specify. Ensure you have selected the correct USB disk before
continuing.
gunzip –d ~/Downloads/caelus.img.gz
Run mount or blkid to find the mount point of the USB drive.
Run umount to unmount the USB drive.
Run gunzip to unzip the disk image:
gunzip –d ~/Downloads/caelus.img.gz
WARNING: dd will overwrite any disk you specify. Ensure you have selected the correct USB disk before
continuing.
Configure the USB boot disk by dragging the FNA kit config JSON file
(<Kit Name>_config.json) to the CONFIG partition on the USB drive.
Connect the USB drive to the USB port on the server labeled ‘fna001’ (see Figure 17).
NOTE: The USB boot disk will configure the entire FNA system. You do not need to connect the USB drive into the
remaining servers.
Facebook Network Appliance Deployment | 27
The remainder of the configuration process is automated. You do not need to interact with the FNA.
You can connect a VGA monitor into the VGA port of ‘fna001’ to watch the configuration
process.
After a few minutes, the host will boot from the USB boot disk and you will see the
FreeBSD boot screen (Figure 18):
The USB boot disk will attempt to access the internet and download software
necessary for the FNA to operate.
Facebook Network Appliance Deployment | 28
If any errors appear on the monitor, take a picture or note the errors seen. Contact the
FNA team (see the Support section) and submit these errors.
Facebook Network Appliance Deployment | 29
URL: https://partners.facebook.com/network/
If you don't have a Facebook account, you need to create one. Go to http://www.facebook.com and follow the
instructions for creating an account. To associate your work email address with your Facebook account,
perform the following steps:
Click the Menu icon ( ) located at the top right of a Facebook page. Click Settings.
Click Email.
Enter your work email address in the New Email field. Click Add.
A confirmation code will be sent to verify the email address. Click Confirm and enter the code. Click
OK.
The portal allows you to authorize others as administrators and set their permissions.
Click the Settings icon ( ) located at the top right of the portal page.
Enter the name and email of the user you wish to make an administrator.
Click Add.
An invite will be sent to the user's email address. Their permission status will remain "pending" until
they accept the invite.
NOTE: Admin access grants access and control to all settings and data within the portal.
NOTE: You can also set the user's permissions to "Read-Only Access". This gives the user complete access to all
portal data, but does not allow access to any settings or controls. To set a user's permission to read-only access,
follow the steps in the Modifying Permissions section and select Read-Only Access.
Click the chevron icon ( ) below the user's email address. Select either Admin Access or Read-Only
Access.
Facebook Network Appliance Deployment | 31
Click Done.
FNA deployments need to connect to your network. To enable this connection, you need to provide your ISP's
Autonomous System Number (ASN). Facebook requires this data to estimate the number of servers your FNA
deployment will require. Facebook also uses this information during the provisioning process to properly route
traffic to the FNA.
Click Add.
Click the Remove link next to the ASN entry you wish to remove.
NOTE: You can enter multiple ASNs. All ASNs should be kept up-to-date.
NOTE: If the ASN entry has a yellow clock icon ( ), it is currently being reviewed by Facebook.
Requesting an FNA kit is a multi-step process that takes several weeks to complete. Facebook evaluates
Facebook Network Appliance Deployment | 33
submissions for qualified ISPs that can utilize an FNA. If your application is approved to receive an FNA,
Facebook will send an FNA kit to the designated ISP address.
Click the Support link at the top of the page. The Support page will appear.
Enter the deployment country where the FNA will be installed. Click Next.
Review the information and click the Begin Order Form button. An online order form will appear in a
new window or tab. Use table 6 as a guideline for order form field entry:
Provide contact information for the This can be the engineering contact.
Shipping primary contact that will receive
Contact
FNA shipments.
Provide contact information for the This can be the engineering contact.
primary contact that will install or
Installation
lead installation of the FNA
Contact
equipment.
Fill out and submit any required order forms located in the Required Forms section near the top of the
order form. Forms that require submission are denoted with the "X" icon ( ). Forms that are currently
pending review are denoted with the "exclamation" icon ( ).
Click Submit if you have completed the order form or Save Draft if you plan to complete the order form
at a later time.
Facebook Network Appliance Deployment | 35
If Facebook selects you to participate in the FNA program, International Business Machines (IBM) will contact
the shipping contact you indicated with further details and will make arrangements to deliver FNA equipment.
IBM is Facebook's logistics partner for FNA equipment.
Facebook Network Appliance Deployment | 36
Once you have installed your FNA equipment and it is brought online, access the Manage FNA to:
The installed kit information page allows you to view the number of servers per cluster, IPv4 and IPv6 network
addresses, advertised prefixes, and disk image. To view this page, follow these steps:
This page displays FNA information sorted by cluster name. You can also request a new kit by clicking
Request a New Kit in the top-right corner.
Facebook Network Appliance Deployment | 37
FNA traffic metrics are located in the Insights page. This page allow you to observe cache-fill and retransmit
information sorted by FNA cluster name. To view traffic information, follow these steps:
Click Insights.
The FNA Traffic information will appear. Each plot represents either "To Subscribers" or cache fill for
each FNA cluster installed.
To view traffic retransmission information, click the button at the top left of the graph. Click FNA
Retransmits. This page will show traffic retransmission as a percentage of the total FNA traffic.
The contact information you provided on the FNA order form can be found in the Peering page. To review this
information, follow these steps:
This page allows you to review your contacts, exchanges, and facilities information.
Facebook Network Appliance Deployment | 38
The contact information you provided on the FNA order form can be found in the Peering page. To review this
information, follow these steps:
This page allows you to view peering session details, including ASN, location, peer type, link speed,
router address, peer port, and status.
Facebook Network Appliance Deployment | 39
If you have issues receiving equipment or installation and configuration issues, use Table 7 to contact the most
appropriate support team for assistance.
NOTE: In some cases, it may be necessary to return the equipment. It is recommended to keep the original
packaging material for two months.
Facebook Network Appliance Deployment | 40
The amount of traffic that the FNA cluster may serve varies based on:
Each server in the cluster is capable of delivering ~4Gbps of traffic. However, depending on the deployment
region, service type (e.g., mobile vs. fixed line), and variance in content type (e.g., images and video), the FNA
cluster may deliver less overall traffic (~3Gbps is expected).
Generally, a high variance in content type will yield lower traffic throughput and low content type variance will
yield higher traffic throughput.
The installation procedure indicates installing the supplied switch directly to our existing router. Our
facility aggregates to our existing switches before connecting to a router. How does this modify the
installation process?
FNA provides aggregate connections from the cache servers directly to the router. However, if you connect the
supplied FNA switch to an intermediary aggregation switch, this configuration is fine and the logical topology
will remain the same.
Yes. FNA is compatible with CGNAT for the RFC 1918 and RFC 6598 address spaces.
Facebook Network Appliance Deployment | 41
I’ve advertised a certain prefix but it doesn’t seem to be served by the FNA. Why is this happening?
Advertising a prefix marks means it as eligible to be served from the FNA, but does not guarantee it will be
served by FNA. There are a few reasons why the FNA may not be serving a prefix which has been advertised
to it:
If it has been less than 24 hours since the FNA has been online or since you have started advertising
the prefix (see Latency Based Routing). Facebook may not have collected enough latency samples in
order to decide whether the FNA cluster will serve the prefix.
If Facebook is observing less latency to the prefix from a nearby Point of Presence (PoP), then the PoP
will be preferred for serving the prefix.
If the system resources (e.g. CPU utilization) have been exhausted.
Which routes should I advertise towards Facebook and how does failover occur between FNA’s?
In most cases, the best option is to announce all customer prefixes. In cases where there may be congestion
from region to region, the best practice is to announce local routes only.
Facebook’s content routing platform measures Round Trip Time (RTT) latency between each Point of
Presence (PoP) and FNA deployment to all prefixes being advertised by a peer ISP. Based on this
measurement and a number of other factors (such as available system resources and link capacity), a client is
routed to the most optimal location for making content requests.
Facebook Network Appliance Deployment | 42
This section provides troubleshooting resources for events that may be encountered during installation or
operation.
Each ProLiant DL380 server and Nexus switch has LEDs that can be used to diagnose the FNA for issues.
These LEDs also useful for verifying whether the FNA was installed correctly and for receiving quick technical
support.
Activity
On, Blinking Green ____________________ There is data activity on the card.
(A)
The drive has sufficient life for
On, Steady Green ____________________
programming and erasing the memory.
The drive has approximately 10% or less
On, Steady Yellow ____________________ life remaining for programming and
Life (L)
erasing the memory.
The drive no longer has program or
On, Steady Red ____________________ erase cycles. Data can be read, but
writing data is at risk.
On, Steady Green ____________________ The drive is operating normally.
A network administrator is toggling the
On, Blinking Green ____________________
LED to locate the server.
Drive warning. One of the following has
Status
occurred:
(S)
On, Steady Yellow ____________________ The maximum temperature threshold
has been reached.
Another issue has occurred.
On, Blinking Red ____________________ A firmware fault code has been issued.
Facebook Network Appliance Deployment | 43
Activity
On, Blinking Green ____________________ There is network activity.
(ACT)
Link A connection exists between the server
On, Steady Green ____________________
(LNK) and the network.
Power
Supply On, Steady Green ____________________ The power supply is on.
In cases where light levels are outside the normal range (between -2dB and -7dB), utilize Table 9 to resolve
the issue.
Table 9: Signal Level Troubleshooting
The signal is too strong. This Fix with your normal operating procedures
Above -2 could be caused by many for strong signals.
reasons.
A fiber optic cable is bent Inspect the fiber optic cable(s) along its
Network connectivity has not length. Gently roll the fiber if you
been established observe bends or kinks.
-21 to -40
All cables are connected to Check that the port connection is
their proper ports, but the enabled
port(s) is disabled Ensure the cables are connected to the
All ports are enabled, but a proper port interface. See Figure 10 for
cable is connected to the port interfaces.
incorrect port
Facebook Network Appliance Deployment | 45
The following are examples of LACP configurations for the most common Cisco IOS and Juniper Junos
routers. Apply the lines of code (below) that are appropriate to your specific router model.
The following configuration settings should be applied to a Juniper Junos router for FNA LACP compatibility:
ae1 {
aggregated-ether-options {
lacp {
active;
periodic fast;
}
}
unit 0 {
description FACEBOOK-FNA;
family inet {
address <address/mask>;
}
family inet6 {
address <address/mask>;
}
}
}
xe-0/0/0 {
description FACEBOOk-FNA;
gigether-options {
802.3ad ae1;
}
}
xe-1/0/0 {
description FACEBOOk-FNA;
gigether-options {
802.3ad ae1;
}
}
To confirm whether the LACP settings were properly applied, run the following commands:
If the following output appears, the LACP settings were properly configured for the Juniper Junos router:
Facebook Network Appliance Deployment | 46
The following configuration settings should be applied to a Cisco IOS-XR router for FNA LACP compatibility:
!
interface Bundle-Ether1
description FACEBOOK-FNA
ipv4 address <address/mask>
ipv6 address <address/mask>
!
interface TenGigE0/0/0/0
description FACEBOOK-FNA
bundle id 1 mode active
lacp period short
!
interface TenGigE0/1/0/0
description FACEBOOK-FNA
bundle id 1 mode active
lacp period short
!
To confirm whether the Cisco IOS-XR LACP was properly configured, run the following command:
If the following output appears, the LACP settings were properly configured on the Cisco IOS-XR router:
Facebook Network Appliance Deployment | 47
The following are examples of BGP peering code for the most common Cisco OOS and Juniper Junos routers.
These examples allow you to configure BGP peering sessions to send Facebook traffic to the FNA cluster.
Contact Facebook (fna-ops@fb.com) if you encounter problems when setting up BGP peering.
NOTE: After the FNA cluster has network access, it will begin downloading application data that enables the FNA
to operate. All application data must be installed before the BPG session is established. The application
data may take some time to download.
The following router BGP configuration should be used for FNA deployments that utilize a Juniper Junos
router:
protocols {
bgp {
group FACEBOOK-FNA-IPv4 {
neighbor <FNA IPv4 address> {
multihop;
family inet {
unicast;
}
export FACEBOOK-FNA-EXPORT;
peer-as 63293;
}
}
group FACEBOOK-FNA-IPv6 {
neighbor <FNA IPv6 address> {
multihop;
family inet6 {
unicast;
}
export FACEBOOK-FNA-EXPORT;
peer-as 63293;
}
}
}
}
policy-options {
policy-statement FACEBOOK-FNA-EXPORT {
term export {
from {
prefix-list FACEBOOK-FNA;
}
then accept;
}
term reject {
then reject;
}
Facebook Network Appliance Deployment | 49
To enable BGP Graceful Restart on a Juniper Junos router, perform the following procedure:
Utilize Juniper’s Configuring Graceful Restart Options for BGP reference document.
Navigate to the appropriate hierarchy level and change the disable statement to enable.
Evaluate whether GR was successfully negotiated by running the following command:
Locate the applicable Junos device and evaluate whether GracefulResart is within the Options
output:
NOTE: Enabling Graceful Restart on the Junos device does not imply that it is in use. Both sides of the session must
support and negotiate Graceful Restart.
The following router BGP configuration should be used for FNA deployments that utilize a Cisco IOX-XR
routers:
!
router bgp x
!
address-family ipv4 unicast
!
address-family ipv6 unicast
!
neighbor <FNA IPv4 address>
remote-as 63293
address-family ipv4 unicast
route-policy FACEBOOK-FNA-EXPORT out
!
neighbor <FNA IPv6 address>
remote-as 63293
address-family ipv6 unicast
route-policy FACEBOOK-FNA-EXPORT out
!
Facebook Network Appliance Deployment | 50
To enable BGP Graceful Restart on a Cisco NX-OS router, perform the following procedure:
If the output string ‘Graceful Restart capability: advertised received’ appears, GR was
successfully negotiated.
Facebook Network Appliance Deployment | 51
The following diagram demonstrates the route of a Facebook request to your ISP both before and after the
FNA deployment:
Facebook Network Appliance Deployment | 52
REMINDER: QSFP+ port 1 is internally multiplexed with the (4) SFP+ uplink ports. The (4) SFP+ ports and the QSFP+
breakout port cannot be utilized at the same time.
Facebook Network Appliance Deployment | 53
The following diagram shows the topology of a (20) server FNA installation:
NOTE: This diagram depicts an FNA cluster with the maximum number of supported servers per switch.
However, when building an FNA cluster beyond eight servers, it is best practice to install the additional servers in
a new rack and maintain equal size clusters (as demonstrated in Section 6.2 Growth Path).
Facebook Network Appliance Deployment | 54
Typical Operating Power 210W with Twinax @ 100% load; (2) PSUs
240W with SR4 optics @ 100% load; (2) PSUs
Maximum Power 340W
Input Voltage by PSU 100V to 240V
Frequency 50-60Hz
Efficiency 89% to 91% @ 220V
Dual Path Power Required
Dual PDUs Required
Typical Operating Power 210W with Twinax @ 100% load; (2) PSUs
240W with SR4 optical transceiver @ 100% load;
(2) PSUs
Maximum Power 340W
Input Voltage by PSU –40V DC to –72V DC
Maximum Current 33A
Efficiency 85% to 88%
Power Supply Efficiency 89% to 91% @ 220V
Dual Path Power Required
Dual PDUs Required
Length 2 Meters
Plug SS10A
Connector HS10S