Active/Active High Availability Considerations

Professional Services - Security Management Framework

Contact Information

Corporate Headquarters:
Palo Alto Networks
3000 Tannery Way
Santa Clara, CA 95054

About This Document

The Operational Enablement documents are designed to enable and inform customers on how to
manage Palo Alto Networks technologies in a consistent and efficient manner. These documents assume
that the reader is already familiar with Palo Alto Networks technology and they are meant to serve as
sections within a runbook on how to manage the platform once deployed.

These documents do not replace other technical documentation published by Palo Alto Networks on their
products and features. For more information about anything referenced in this document, see the
technical documentation found at:
https://www.paloaltonetworks.com/documentation

© 2019 Palo Alto Networks, Inc. Palo Alto Networks is a registered trademark of Palo Alto Networks. A list of our
trademarks can be found at http://www.paloaltonetworks.com/company/trademarks.html. All other marks mentioned
herein may be trademarks of their respective companies.

2
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 Table of Contents
About This Document ................................................................................................................................. 2
High Availability Overview ......................................................................................................................... 5
HA Overview ................................................................................................................................................ 5
Active/Active Concepts .............................................................................................................................. 6
Active/Active Architectures ....................................................................................................................... 7
Virtual Wire ................................................................................................................................................ 7
Layer 3 ...................................................................................................................................................... 7
Flow Asymmetry........................................................................................................................................ 7
Split Data Center Considerations .............................................................................................................. 7
HA Links and Backup Links ....................................................................................................................... 8
HA1 - Control Link ..................................................................................................................................... 8
HA2 - Data Link ......................................................................................................................................... 8
Backup Links ............................................................................................................................................. 8
HA3 - Packet-Forwarding Link .................................................................................................................. 8
Active/Active Configuration ....................................................................................................................... 8
General Configuration ............................................................................................................................... 8
HA2 Considerations .............................................................................................................................. 8
HA Timers ................................................................................................................................................. 8
Link and Path Monitoring .......................................................................................................................... 9
Link Monitoring ...................................................................................................................................... 9
Path Monitoring ..................................................................................................................................... 9
Configuration Considerations .................................................................................................................... 9
HA3 Interface ........................................................................................................................................ 9
Configuration Sync ................................................................................................................................ 9
Session Processing ............................................................................................................................... 9
Floating IPs ........................................................................................................................................... 9
ARP Load Sharing ................................................................................................................................ 9
Failover Performance ............................................................................................................................... 10
Impacts to Services .................................................................................................................................. 10
Network Address Translation Concepts .................................................................................................. 10
Session State Considerations ................................................................................................................. 10
Session Synchronization ......................................................................................................................... 10
TCP State Checking ................................................................................................................................ 10
Fragmented Packets ............................................................................................................................... 10
Troubleshooting ........................................................................................................................................ 10
Global Counters ...................................................................................................................................... 10

3
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 Session Details ....................................................................................................................................... 11
Active/Active Checklist ............................................................................................................................. 12

4
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
High Availability Overview
High availability (HA) is a deployment in which two firewalls are placed in a group and their configuration
is synchronized to prevent a single point of failure on your network. A heartbeat connection between the
firewall peers ensures seamless failover in the event that a peer goes down. Setting up two firewalls in an
HA pair provides redundancy and allows you to ensure business continuity.

Improves Security Improves Manageability

Improves Performance Improves Availability X
Customized Benefit/Gain:

Suggested RACI Role Security Activity Frequency (i.e., N/A

(generic suggestion Engineer/Administrator daily, weekly, monthly,
which will vary by etc.)
organization)
Required Skills: PAN-OS Configuration, L2 and L3 Networking

HA Overview
HA allows you to minimize downtime by making sure that an alternate firewall is available in the event
that the peer firewall fails. The firewalls in an HA pair use dedicated or in-band HA ports on the firewall to
synchronize data - network, object, and policy configurations - and to maintain state information. Firewall-
specific configuration such as management interface IP address or administrator profiles, HA specific
configuration, log data, and the Application Command Center (ACC) information is not shared between
peers. For a consolidated application and log view across the HA pair, you must use Panorama, the Palo
Alto Networks centralized management system.
Active/Active High Availability solution from Palo Alto Networks complements the Active/Passive solution
and was designed for customers that require redundant, active network paths through the firewall system.
Palo Alto Networks firewalls support stateful Active/Passive or Active/Active high availability with session
and configuration synchronization with a few exceptions:
• The PA-200 firewall supports HA lite only. HA lite is an Active/Passive deployment that provides
configuration synchronization and some runtime data synchronization such as IPSec security
associations. It does not support any session synchronization (HA2), and therefore does not offer
stateful failover.
• The VM-Series firewall in AWS supports Active/Passive HA only; if it is deployed with Amazon
Elastic Load Balancing (ELB), it does not support HA (in this case ELB provides the failover
capabilities). The active peer continuously synchronizes its configuration and session information
with the identically configured passive peer. A heartbeat connection between the two devices
ensures failover if the active device goes down. When the passive peer detects this failure, it
becomes active and triggers API calls to the AWS infrastructure to move all the dataplane
interfaces (ENIs) from the failed peer to itself. The failover time can vary from 20 seconds to over
a minute depending on the responsiveness from the AWS infrastructure. The peers must be
deployed in the same AWS availability zone.
• The VM-Series on Azure does not support the traditional Active/Passive high availability with
session synchronization, as it is not a good fit for the public cloud architecture where you don’t
need to protect against hardware failures (CPU, memory, NIC, HDD) like you would in a private
cloud deployment. Instead, for both small and large deployments, use a scale out architecture

5
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
 using cloud-native load balancers such as the Azure Application Gateway or Azure Load
Balancer to distribute traffic across a set of healthy instances of the firewall.
• On Google cloud platform it is common practice to use a scale-out architecture rather than larger,
higher performing VMs. This architecture (sometimes called a sandwich deployment) avoids a
single point of failure and enables you to add or remove firewalls as needed.

Active/Active Concepts
Active/Active mode requires advanced design concepts that can result in more complex networks. Both
firewalls in the pair are active and processing traffic and work synchronously to handle session setup and
session ownership. Both firewalls individually maintain session tables and synchronize to each other.
Active/Active HA is supported only in virtual wire and Layer 3 deployments.
In Active/Active HA mode, the firewall does not support DHCP client. Furthermore, only the Active-
Primary firewall can function as a DHCP Relay. If the Active-Secondary firewall receives DHCP broadcast
packets, it drops them.
An Active/Active configuration does not “load-balance” traffic. Although you can load-share by sending
traffic to the peer, no load balancing occurs. Ways to load share sessions to both firewalls include using
ECMP/Dynamic Routing, multiple ISPs, and external load balancers. The recommend deployment is to
use external load balancers.
The two requirements where Active/Active HA configurations should be considered are:
• Network designs where flow asymmetry exists.
• Network designs where it is necessary to have routing protocol adjacency with both devices in the
cluster concurrently.
The Active/Active HA feature was explicitly designed to address these use cases. Other goals, such as
increased performance or throughput, cannot be realized. Depending on how you implement
Active/Active HA, it might require additional configuration such as activating networking protocols on both
firewalls, replicating NAT pools, and deploying floating IP addresses to provide proper failover. Because
both firewalls are actively processing traffic, the firewalls use additional concepts of session owner and
session setup to perform Layer 7 content inspection. Active/Active mode has faster failover and can
handle peak traffic flows better than Active/Passive mode because both firewalls are actively processing
traffic.

Note: In Active/Active mode, the HA pair may temporarily process more traffic than what one firewall
can normally handle. This must not be a design goal, however, because a failure of one firewall causes
all traffic to be redirected to the remaining firewall in the pair. The design must allow the remaining
firewall to process the maximum capacity of traffic loads under “worst-case” conditions. If the design
oversubscribes the capacity of the remaining firewall, high latency and/or application failure can occur.
If the potential for flow asymmetry does not exist in your environment, an Active/Passive deployment is
the recommended deployment.

6
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Active/Active Architectures
In general, Active/Active architectures should employ design symmetry. Use the same physical interfaces,
Zone, and Virtual Router names on both devices.

Virtual Wire
Recommended for a Layer 3 topology only. Implementing Active/Active virtual wire in a Layer 2 topology
can result in a bridge loop if loop-avoidance protocols are not used in the surrounding Layer 2
environment.
• Flow asymmetry should be removed by forcing all inbound and outbound traffic through one
firewall.
• All routers to prefer routes learned via the same side routers.
• All routers use BFD on dynamic routing protocols for rapid failure detection of vwire path.
• Firewalls do not participate in dynamic routing protocols or influence routing decisions on
upstream/downstream devices as this would increase failover time.

Layer 3
In a Layer 3 architecture, upstream and downstream routers have a Layer 3 adjacency with the firewalls.
Each Layer 3 interconnect is a unique subnet facilitated by Layer 3 interfaces on the Palo Alto Networks
devices. There may be intermediate Layer 2 devices between the Layer 3 devices and the firewalls.
In this solution, the firewalls can participate in dynamic routing protocols and may influence routing
decisions on upstream / downstream devices.
If using a dynamic routing protocol, do not enable VR Synchronization.

Flow Asymmetry
A primary design goal in any Active/Active architecture should be to reduce or eliminate the opportunity
for flow asymmetry. Flow asymmetry results in complex packet handling requirements by the firewalls and
can lead to increased application latency or flow handling errors.
Dynamic routing protocol metrics should be used to prefer a single/same device for both client to server
(C2S) and server to client (S2C) flows.
In a statically routed Layer 3 deployment, a floating IP can be used to send traffic to a single device. This
assumes that both the Active-Primary and Active-Secondary devices share a common Layer 2 segment
with the upstream/downstream Layer 3 devices.

Split Data Center Considerations

In a split data center architecture, the following factors must be considered:
• Latency: No more than 20ms is recommended.
• HA2 latency must be less than or equal to HA3 latency.
• If the HA2 or HA3 connection traverses a switch, jumbo frames must be enabled.
• If the HSCI interface is used for HA3, it must be a Layer 1 connection.

Note: As with any high availability architecture, the entire system must be evaluated to ensure the
desired performance is achieved.

7
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
HA Links and Backup Links
HA1 - Control Link
The HA1 link is used to exchange hellos, heartbeats, and HA state information, and management plane
sync for routing and User-ID information. The firewalls also use this link to synchronize configuration
changes with its peer. The HA1 link is a Layer 3 link and requires an IP address.

HA2 - Data Link

The HA2 link is used to synchronize sessions, forwarding tables, IPSec security associations and ARP
tables between firewalls in an HA pair. Data flow on the HA2 link is always unidirectional (except for the
HA2 keep-alive); it flows from the active or Active-Primary firewall to the passive or Active-Secondary
firewall. The HA2 link is a Layer 2 link, and it uses ether type 0x7261 by default.

Backup Links
Provide redundancy for the HA1 and the HA2 links. In-band ports are used as backup links for both HA1
and HA2. Consider the following guidelines when configuring backup HA links:
• The IP addresses of the primary and backup HA links must not overlap each other.
• HA backup links must be on a different subnet from the primary HA links.
• HA1-backup and HA2-backup ports must be configured on separate physical ports.

HA3 - Packet-Forwarding Link

The firewalls use this HA3 link for forwarding packets to the peer during session setup and asymmetric
traffic flow. The aggregate interfaces can also provide redundancy for the HA3 link; you cannot configure
backup links for the HA3 link.

Active/Active Configuration
General Configuration
Dedicated HA interfaces should be used where available. Backup HA interfaces for the HA1 and HA2
interfaces should be provisioned to ensure service redundancy. A Group ID for the pair, must be the
same for both firewalls. The firewall uses the Group ID to calculate the virtual MAC address (range is 1-
63). Two different pairs in the same Layer 2 domain must not have the same Group ID.
HA2 Considerations
Failure Conditions
“Split data path” is the recommended setting for an Active/Active configuration and will allow each device
to “take ownership” of its sessions in the event of a failure of the HA2 service.
“Log only” is appropriate for Active/Passive deployments.
Bandwidth Considerations
Design adequate bandwidth for the HA2 interface.

HA Timers
High Availability timers facilitate a firewall to detect a firewall failure and trigger a failover. To reduce the
complexity in configuring HA timers, you can select from three profiles: Recommended, Aggressive, and
Advanced. These profiles auto-populate the optimum HA timer values for the specific firewall platform to
enable a speedier HA deployment.

8
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Link and Path Monitoring
Link Monitoring
In an Active/Active configuration, it is often desirable to not have link failure events trigger a tentative
state and transition of the Active-Primary role.
Path Monitoring
Path Monitoring monitors the full path through the network to IP addresses identified as “critical” to the
success of the network path external to the firewall system. ICMP pings are used to verify reachability of
the IP address.

Configuration Considerations
HA3 Interface
Use HSCI interfaces when available and AE for others. If there are intermediate switches in the HA3
transport path, jumbo frames must be configured. AE interfaces cannot be used for increased aggregate
bandwidth for HA3 service because the session between peers is hashed to a single physical interface.
AE is used for transport redundancy only. Consider reducing packet-passing on the HA3 interface by
reducing flow asymmetry or configuring Session Owner and Session Setup as First Packet.
Configuration Sync
Enable the VR sync option when the virtual router is not configured for dynamic routing protocols.
Enable the QoS sync option when both peers have similar link speeds and require the same QoS profiles.
Session Processing
In an HA Active/Active configuration, both firewalls are active simultaneously, which means packets can
be distributed between them. Such distribution requires the firewalls to fulfill two functions:
• Session Ownership - recommended setting is First Packet
• Session Setup - recommended setting is IP Modulo
Typically, each firewall of the pair performs one of these functions, thereby avoiding race conditions that
can occur in asymmetrically routed environments.
Floating IPs
In a Layer 3 deployment of HA Active/Active mode, you can assign Floating IP Addresses (FIP), which
move from one HA firewall to the other if a link or firewall fails. Floating IP addresses are recommended
when you need functionality similar to Virtual Router Redundancy Protocol (VRRP). Floating IP
addresses can also be used to implement VPNs and source NAT, allowing for persistent connections
when a firewall offering those services fails.
ARP Load Sharing
In a Layer 3 interface deployment and Active/Active HA configuration, ARP load-sharing allows the
firewalls to share an IP address and provide gateway services. Use ARP load-sharing only when no
Layer 3 device exists between the firewall and end hosts. For example, when end hosts use the firewall
as their default gateway.

9
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Failover Performance
When deploying Palo Alto Networks firewalls in an HA cluster, there are some considerations to achieve
optimal failover times. The total failover time depends on several additional factors as well as the HA
timer tunings available in the system configuration.
In general, failover performance is the sum of:
• Failure detection
• Failover time
• Network reconvergence
Configuration optimization in these three areas will achieve the best possible failover performance.

Impacts to Services
Network Address Translation Concepts
Network Address Translation (NAT) in an Active/Active configuration has additional considerations due to
the requirement that both systems are active on the network.
In an Active/Active HA configuration:
• You must bind each Dynamic IP (DIP) NAT rule and Dynamic IP and Port (DIPP) NAT rule to
either Device ID 0 or Device ID 1.
• You must bind each static NAT rule to either Device ID 0, Device ID 1, both Device IDs, or the
firewall in the Active-Primary state.

Session State Considerations

Session Synchronization
ICMP, Host, and Multicast sessions are not synchronized in an Active/Active setup.

TCP State Checking

Even in the presence of flow asymmetry, it should not be necessary to allow Non-SYN TCP traffic for
proper session handling to occur. Use of IP Modulo for Session Setup will also help mitigate issues with a
device not properly handling a packet that is part of an existing session not owned by the receiving
device.

Fragmented Packets
All fragments must be handled by the same device.

Troubleshooting
This section documents troubleshooting commands that are relevant to the Active/Active deployment.

Global Counters
There are several useful global counters that are relevant to the Active/Active deployment. Run the
show counter global filter aspect aa command to view them.
Global counters can also be used to troubleshoot suspected TCP state issues such as improper handling
of NON-SYN TCP packets.

10
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Session Details
The session detail information has fields that are relevant in troubleshooting Active/Active session
handling:
• Owned by local device
• Setup by local device
• Updated by peer

11
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.
Active/Active Checklist
 The same model - The firewalls in the pair must be of the same hardware model.
 The same PAN-OS version - The firewalls must be running the same PAN-OS version and must
each be up-to-date on the application, URL, and threat databases.
 The same multi-virtual system capability - Both firewalls must have multi-virtual system capability
either enabled or not enabled. When enabled, each firewall requires its own multiple virtual
systems licenses.
 The same type of interfaces - Dedicated HA links, or a combination of the management port and
in-band ports that are set to interface type HA.
 The HA interfaces must be configured with static IP addresses only, not IP addresses obtained
from DHCP (except AWS can use DHCP addresses). Determine the IP address for the HA1
(control) connection between the HA peers. The HA1 IP address for the peers must be on the
same subnet if they are directly connected or are connected to the same switch.
 For firewalls without dedicated HA ports, you can use the management port for the control
connection. Using the management port provides a direct communication link between the
management planes on both firewalls. However, because the management ports will not be
directly cabled between the peers, make sure that you have a route that connects these two
interfaces across your network.
 If you use Layer 3 as the transport method for the HA2 (data) connection, determine the IP
address for the HA2 link. Use Layer 3 only if the HA2 connection must communicate over a
routed network. The IP subnet for the HA2 links must not overlap with that of the HA1 links or with
any other subnet assigned to the data ports on the firewall.
 Each firewall needs a dedicated interface for the HA3 link. The PA-7000 Series firewalls use the
HSCI port for HA3. The PA-5200 Series firewalls can use the HSCI port for HA3 or you can
configure aggregate interfaces on the dataplane ports for HA3 for redundancy. On the remaining
platforms, you can configure aggregate interfaces on dataplane ports as the HA3 link for
redundancy.

12
Active/Active HA Considerations Proprietary and Confidential ©2019 Palo Alto Networks, Inc.