IPSec Standards and Protocols

AH and ESP
Matt Norris
 IPSec Standards and Protocols
AH and ESP
IPSec is one of several VPN standards that have allowed secure, low-cost connectivity
and data transport between locations over unsecured communication lines. Although
IPSec is most commonly used for the Internet, its use has been extended to secure
communications within local area networks between client and server and server to
server.
When designing and implementing a VPN solution, it is important to understand that
IPSec is not a single protocol but is comprised of many protocols that can be combined to
provide varying levels of protection. The premise of choosing and combining different
protocols makes IPSec extremely flexible and manageable if the implementer
understands the primary protocols and connection modes used in the IPSec standard.
The two protocols that, individually or in tandem, form the backbone of IPSec are
Authentication Header AH) and Encapsulating Security Payload (ESP). The two modes
that an IPSec connection, known as Security Association (SA), can operate in are Tunnel
Mode or Transport Mode.

What Is a SA (Security Association)?

The SA is the “connection” between the IPSec peers that is required for secure data
exchange and is configured for either Transport Mode or Tunnel Mode. It is composed of
the negotiated parameters that will be used for data handling. SA parameters include the
IP address of the peer; encryption algorithm; protocol format (AH or ESP); and security
parameter index (SPI), a unique number assigned to each SA and used to manage
multiple SAs. Once negotiated, each peer stores the SA parameters locally in a SA
database (SAD). The SA is unidirectional, meaning a separate SA is needed for each
inbound and outbound connection and bidirectional traffic therefore requires two SAs.

IPSec Protocols: AH and ESP

AH (Authentication Header) is a format protocol defined in RFC 2402 that provides data
authentication, integrity, and nonrepudiation but does not provide data confidentiality.
AH adds security to the communication stream by encrypting nonvolatile fields of the IP
header and creating a message digest value at origination that can be re-created and
compared at the destination. The message digest value is created through the application
of a one-way hash of the IP header and data portion of the packet. After that message
digest value is created, an encrypted AH header is inserted between the original IP header
and data portion of the packet. The encrypted AH header includes the message digest
value and authentication information for the packet. The IP packet is then sent to the
IPSec peer. The IPSec peer hashes the IP header and data portion of the packet to create a
message digest value and compares it to the hash value in the AH header that it has
decrypted. If the message digest values match, it ensures that the packet has not been
modified in transit. If a packet is determined to have been modified in transit, it is
rejected. AH can be utilized in either Transport Mode or Tunnel Mode because it protects
the outer IP header whether it is the original header in Transport Mode or a new header
created in Tunnel Mode.
AH Process

Protected IP Header and AH Header

ESP is a format protocol defined in RFC 2406 that provides data confidentiality (through
encryption) and is typically what we think of when deploying a “secure” VPN solution.
ESP can optionally provide integrity and data origin authentication through the use of a
hash and can provide replay attack protection. ESP adds security to the communication
stream by encrypting the data payload when in Transport Mode or by encrypting and
encapsulating the entire IP packet when in Tunnel Mode as described in the following.
ESP supports the use of symmetric encryption algorithms, including DES, 3DES, and
AES, for confidentiality and the use of MD5 HMAC and SHA1 HMAC for data
authentication and integrity. Similar to AH authentication and integrity, ESP creates a
hash at the point of origination that can be compared at the destination. But unlike AH,
ESP cannot protect the unencrypted IP header--which is why ESP and AH are commonly
combined to add another level of protection.
IPSec Connection Modes: Transport and Tunnel
Transport Mode: Protection is provided for the data in the IP packet through encryption
but not for the IP header information, which remains unchanged. Transport Mode adds
only a few bytes of information to each IP packet, in the form of an IPSec header, and it
allows for quality-of-service (QoS) management on the network. Transport Mode is
typically used when end-to-end encryption is required and supported by the peers and is
deployed between or within locations.

Transport Mode IP Packet (Utilizing ESP)

Typical Transport Mode Data Exchange

Tunnel Mode: Protection is provided for the entire IP packet, which is encrypted and
then encapsulated in a new IP packet including a new IP header and an IPSec header.
Tunnel Mode is typically used on IPSec gateway devices such as firewalls, routers, and
VPN appliances connecting remote locations such as branch offices. The gateway acts as
an IPSec proxy for the clients that are located behind the device. Clients forward IP
packets to the gateway in the clear. The gateway device then encrypts the packet and
forwards it to an IPSec peer, which in turn decrypts the packet and forwards it to the
destination client.

Tunnel Mode IP Packet (Utilizing ESP)

 Original Packet
IP Header DATA

New IPSec IP Header

DATA (Encrypted) Tunnel Mode Packet
IP Header Header (Encrypted)

Typical Tunnel Mode Data Exchange

After the protocols (AH and ESP) and the modes of transportation (Transport or Tunnel)
are understood, designing a secure communication stream can become a more
manageable task. Of course, these steps are only a few in the overall process and the
architect of the secure communication design should continue to understand all phases
and processes involved in VPN creation and its ongoing maintenance.