DO NOT REPRINT  Routing

© FORTINET

In this lesson, we are going to talk about how to route traffic with FortiGate devices.
DO NOT REPRINT  Routing

© FORTINET

After completing this lesson, you should have these practical skills that you can use
to implement routing failover and load balancing using static routes. You will also
learn how to configure link aggregation, policy routes, and black hole routes. Finally,
you will learn some debug commands for troubleshooting routing problems.
Although this lesson briefly introduces the concept of dynamic routing, it is mostly
about implementing routing with static and policy-based routes.

Lab exercises can help you to test and reinforce your skills.
DO NOT REPRINT  Routing

© FORTINET

What is routing?

Routing decides where FortiGate in NAT mode will send the packets that it receives,
and that it generates. A routing table contains routing rules. For example, FortiGate
can check the destination field of the packet’s IP header. If routing rules match that
destination, FortiGate can transmit the packet from port1 to port2, towards Router 1.

If an allowed packet is not destined for the FortiGate itself — not administrative
access, for example — FortiGate must relay the packet. FortiGate searches for
matching routes in the routing table that it can use to deliver the packet. FortiGate
either delivers the packet directly to its final destination, or relays it to the next router
along the path towards the destination.

Usually, IP routing is done by taking into account only the destination IP address.
However, as we’ll see later, you can decide to route packets using more than just
that.

Proper routing configuration is important. If the routing directions are misconfigured,

packets will not reach their destination and will be lost.
DO NOT REPRINT  Routing

© FORTINET

One type of manually configured route is called a static route. In the route table, its
“Type” column is “Static”.

We are manually telling the FortiGate device, “When you see a packet whose
destination is within this specific range of destination addresses, send it through this
network interface, towards this router.” We also configure the distance and priority
so that FortiGate knows which routes to load into memory, and in what order. We
will talk about distance and priority in later slides.

For example, in simple home networks, DHCP automatically retrieves and

configures one static route. Your modem then sends all outgoing traffic through your
ISP’s Internet router, which can relay packets to their destination.

When do you not require a static route?

When a destination is cabled directly to one of FortiGate’s network interfaces, with

no router in between, FortiGate will be aware of the destination. In the route table,
its “Type” is “Connected”.
DO NOT REPRINT  Routing

© FORTINET

For large networks, manually configuring hundreds static routes may not be
practical.

Your FortiGate can help, by configuring routes automatically. FortiGate supports

several dynamic routing protocols:
RIP, OSPF, BGP, and IS-IS.

In dynamic routing, FortiGate communicates with nearby routers to discover their

paths, and to advertise its own directly connected subnets. Discovered paths are
automatically added to FortiGate’s routing table. (So verify that your neighbor
routers are trusted and secured!)

Larger networks also may need to balance routing load among multiple valid paths,
and detect and avoid routers that are down. We’ll discuss that soon also.
DO NOT REPRINT  Routing

© FORTINET

Which rows are “extra” – automatic entries that aren’t from your static routes
configuration?
• Directly connected subnets – When a subnet is assigned to a FortiGate’s
interface, a route to the subnet is automatically added to the routing table. The
FortiGate knows how to route those packets.
• Dynamic routes – On larger networks, your FortiGate may receive routes from
other routers, via protocols such as BGP. This is faster and more scalable than
manually configuring many routers.

Which configured routes aren’t loaded into this table?

• Worse routes to the same IP – Only the best paths should be used. We will see
in a later slide how the best path is elected when there are multiple routes to the
same destination.
• Policy routes – These are omitted, too. Why? By design, policy routes override
the routing table – we don’t want them to be ignored, losing precedence to OSPF
or static routes. So they have to be in a separate table, which is searched before
this one. We’ll discuss policy routes later.

So remember, expect differences from your configured list of static routes. And
when troubleshooting, don’t only check this table. Also check the table for policy-
based routes, and (if you’re using dynamic routing) your other routers.
DO NOT REPRINT  Routing

© FORTINET

In the routing table, each of the entries has a few pieces of data, such as distance
and gateway IP. They are used to relay or deliver each matching packet.

Destination IP addresses and gateway routers are self-explanatory. The device is

the name of the outgoing interface where the packet will be routed to. But what
about the distance, metric, and priority? How do they effect which routing path
packets will use?

Let’s explain each briefly.

DO NOT REPRINT  Routing

© FORTINET

Distance, or administrative distance, is a number that estimates the reliability or

quality of each routing protocol and static route. If there are two routes to the same
destination, the one with the lower distance is added or loaded to the routing table,
as it is considered to be more reliable.
By default, for example, routes learned via the RIP protocol have a higher distance
that routes learned via the OSPF protocol, as OSPF is considered to be more
accurate than RIP.
DO NOT REPRINT  Routing

© FORTINET

In the case of routes learned via a dynamic routing protocols, metric is another
element that is used to determine the best route to a destination. If two routes have
the same distance, the metric is then used for tie breaking. The route with the
lowest metric is loaded to the routing table.

How the metric is measured depends on the routing protocol. RIP uses hop counts:
how many routers must be used to reach the destination. OSPF uses cost, which is
determined by how much bandwidth a link has.
DO NOT REPRINT  Routing

© FORTINET

In the case of static routes, the priority is used for tie breaking when the distances
are the same. FortiGate will use the route with the smallest number configured in
the route’s priority setting.

In other words, if we have two routes with the same distance to the same
destination, only the one with the smallest priority will be used. Note that unlike with
distances/metrics, both routes with the same distance are loaded into the routing
table. However, only the route with the smallest priority will be routing traffic. This,
as we will see later, is an important concept when dealing with reverse forwarding
path check issues.
DO NOT REPRINT  Routing

© FORTINET

This is summary of the logic behind which routes are loaded into the routing table.

Routes are only active if the interface is currently both physically linked and
administratively “up.” If the cable isn’t plugged in, or if a Wi-Fi network has no signal,
for example, packets can’t be transmitted along that path. All routes through that link
will be temporarily unloaded from the table until the link is available again.

When 2 or more actives routes have the same destination subnet, only the one with
the smallest distance is loaded to the routing table.

If the distances are equal, only the routes with the smallest metric are included.

If the metric also is identical, then, depending on the dynamic routing protocol’s
rules, FortiGate will select which one to include in the routing table.
DO NOT REPRINT  Routing

© FORTINET

Static routes are simple, and are often enough for small networks. Policy routes,
however, are more powerful. They can match more than just the destination IP
address. An example? If you have two links – a slow one and a fast one – you can
route packets from low-priority source IPs to the slow link.

Policy routes with the action forward traffic have precedence over static and
dynamic routes. So, if a packet matches the policy route, the FortiGate bypasses
the routing table lookup.

Like static routes, policy routes must be valid: a destination and gateway are
required, and disconnected or “down” links can’t be used. For policy routes, though,
packets also must match all subnets, ToS bits, and port numbers that you specify.
So if a setting shouldn’t be a criteria for matching, leave it blank.
DO NOT REPRINT  Routing

© FORTINET

When a packet matches a policy route, the FortiGate takes either one of two
actions. Either it routes the packet to the configured interface and gateway,
bypassing the routing table; or it stops checking the policy routes, so the packet will
be routed depending on the routing table.
DO NOT REPRINT  Routing

© FORTINET

Many aspects of FortiGate are (at least by default) stateful, so it decides many
things at the beginning of a session, when it receives the first packets.

For each session, FortiGate makes only 2 routing lookups:

1. Upon the first packet sent by the originator, and
2. Upon the first reply packet coming from the responder.
After that, FortiGate writes the routing information to its session table. Subsequent
packets are routed according to the session table, not the routing table. So all
packets that belong to the same session follow the same path, even after a change
in the static routes. There is an exception to this rule, though: if FortiGate
detects a change in the OSPF network topology, it removes route information for the
session table, and then FortiGate makes another routing table lookup to rebuild the
routes in the session table.
DO NOT REPRINT  Routing

© FORTINET

How does FortiGate decide routes? FortiGate has multiple routing modules. This
diagram shows the logic among them.

First FortiGate searches its policy routes. You can view them with the command
“diagnose firewall proute list”. If there is a match in the policy routes
and the action is Forward Traffic, FortiGate will use the policy route. If the action is
Stop Policy Routing, the FortiGate will use the next table.

After that, FortiGate searches its route cache. You can view that with the CLI
command “diagnose ip rtcache list”. If a match exists, the packet is sent
to that next-hop gateway.

Finally, FortiGate searches the forwarding information base (FIB). The FIB is
generated by the routing process, and is the table used for packet forwarding. Think
of the routing table’s purpose as for management, while the FIB is for forwarding.
This separation becomes more clear in FortiGate active-active HA. In an HA cluster,
both route management and forwarding tables exist on the master FortiGate. But on
the slave FortiGate, only the forwarding table exists.

If there’s no match in any of those tables, FortiGate will drop the packet because it
is unroutable.
DO NOT REPRINT  Routing

© FORTINET

We saw how the distance, metric and priority are used to determine the best route
to a destination. So, what happens when two or more routes to the same destination
share the same values for those routing elements?

If the routes are static, OSPF or BGP, FortiGate balances the traffic among all the
routes. This is what is called Equal Cost Multi-path (ECMP).
DO NOT REPRINT  Routing

© FORTINET

When the FortiGate is doing ECMP, one of the these four methods is used.

Sessions can be balanced among equal routes depending on the source IP

address, source and destination IP addresses, or interface weight. There is an
additional method called spillover, where the FortiGate will use a primary route until
a traffic volume threshold is reached; after that, another route will be used.
DO NOT REPRINT  Routing

© FORTINET

(slide contains animation)

This is an example of ECMP. In the FortiGate routing table, there are two default
routes with the same distance and priority. One using the wan1 interface, another
one using the wan2 interface. So, outgoing traffic is load balanced among the two
ISPs.

If an interface becomes unavailable because of, for example, a physical

disconnection, all routes associated with that link/gateway are temporarily removed
from the routing table.

(click)

So if WAN1 went down, its routes would be dropped from the routing table. The only
remaining available default route for traffic would be through WAN2.
When WAN1 comes up again, then its routes will be loaded back into the routing
table.
DO NOT REPRINT  Routing

© FORTINET

(slide contains animation)

If you do not want to load balance, you can change which route will be primarily
used for the outgoing traffic by changing the priority number.

(click)

In this way, FortiGate will simply switch to use the route with the smallest priority.
Remember that both routes are still in the routing table, as long as they both keep
the same distance number.
DO NOT REPRINT  Routing

© FORTINET

Link health monitor is a mechanism for detecting when a router along the path is
down. It is often used where there are redundant routers onsite, such as in HA
deployments, or for dual ISP links.

When configured, FortiGate periodically sends signals through one of the gateways
to a server that acts as a beacon. The server can be any host that should normally
be reachable via that path. Usually, it’s best to choose a stable server with robust
infrastructure, and to choose the protocol that the server would normally respond to.

If the FortiGate stops receiving a replay from the server, all the routes using that
gateway will be removed from the routing table. Alternatively, you can configure the
unit to administratively bring down an interface, so all routes using that interface will
be removed. While a server is unresponsive, FortiGate will continue to send link
health monitor signals. As soon as FortiGate receives a reply, it will reinstate the
routes.

It may be useful to choose a server that is indirectly attached, located 1 or 2

hops beyond the FortiGate’s gateway. This does not exactly test availability of
this one gateway, but rather the combination of gateways. That way, the
FortiGate will accurately indicate availability of services and subsequent hops.
DO NOT REPRINT  Routing

© FORTINET

Here is where you configure the link health monitor.

You must enter the egress interface, the IP address of the gateway router, and the
IP address and the protocol (HTTP, ICMP, UDP or TCP) of a beacon that is beyond
that gateway.
DO NOT REPRINT  Routing

© FORTINET

Packets are sometimes dropped for reasons where routing and security are related.

Reverse path forwarding (RFP) is a mechanism that protects the FortiGate

and the network from IP spoofing attacks. It checks if there is a valid route back
to the packet’s source, through the interface where the packet is coming from. If
there is not a valid route, the packet is dropped.

This checking is executed over the first packet of any new session. It is also
executed after a route change, over the next packet in the original direction.

When packets are dropped because of the RFP mechanism, the debug flow will
output an error like the one shown in this slide.
DO NOT REPRINT  Routing

© FORTINET

(slide contains animation)

Here’s a sample network setup and routing table.

There are two routing errors here, two interfaces that won’t route traffic properly.
They are port1 and wan2.

port1 will not route traffic properly. The reason is because of the subnet for the
computers. They’re in 10.0.0.0/24, and there’s no route for that subnet in the routing
table to egress through port1.

(click)

So anything coming from 10.0.0.0/24 to that interface will be dropped because that
subnet cannot be routed back.
DO NOT REPRINT  Routing

© FORTINET

The problem is fixed by adding a route to 10.0.0.0/24. Now, when FortiGate does
the RPF check for the incoming packet, it finds a valid route to that subnet through
out port1. The packet is now accepted.
DO NOT REPRINT  Routing

© FORTINET

The other interface that will not be able to route traffic properly either is WAN2.
While it is physically connected to the Internet, the only IP addresses that would be
valid as sources or destinations would be those in the 2.2.2.0/30 subnet. So,
incoming Internet traffic will not pass the RPF check and will be dropped.
DO NOT REPRINT  Routing

© FORTINET

(slide contains animation)

Once again, this is fixed by adding a route for wan2. In this case, the route needs to
act as a default gateway in order to provide Internet access. To become part of
the routing table, it needs to have the same distance as the default route for
wan1. They both can have different priorities, but as we saw in previous slides, they
must have the same distance to be included in the routing table.

(click)

If the priorities are also the same, this creates a situation like the one we saw for
ECMP. So, if the destination is the Internet, there are 2 possible paths to take:
through either wan1 or wan2. Some sessions will exit from wan1, and others will
exit from wan2.
DO NOT REPRINT  Routing

© FORTINET

Reverse path forwarding can be either strictly or loosely enforced.

Loose RPF checks that the sender can be routed out from the interface where the
packet was received. This simply confirms that a response is possible.

Strict RPF requires that the receiving interface is not only valid, but that it is also
the best interface for the reply. If you have multiple routes, it must be the preferred
one.
DO NOT REPRINT  Routing

© FORTINET

(slide contains animation)

Let’s look at an example of loose RFP.

(click)

In this case, 20.20.20.20 pings 10.10.10.5, but fakes a source IP of 10.10.10.6,

making the packet appear to be initiated from the internal network. Loose RPF
would allow this traffic because the route on wan1 is a default route (0.0.0.0/0),
which is valid (although not the best one).

(click)

What would happen next is that 10.10.10.6 would send the SYN/ACK packet to the
“real” device with the IP address 10.10.10.5.

(click)

But since 10.10.10.5 is not expecting SYN/ACK packets (because it has not
previously sent any SYN packet to 10.10.10.6), it will reply with a TCP Reset (RST)
packet.
DO NOT REPRINT  Routing

© FORTINET

(slide contains animation)

Let’s see what happens in the same topology with strict reverse path forwarding.

(click)

Strict RPF drops the packet. The default route in wan1 is a valid route to the subnet
10.10.10.0/24, but it not the best route. The best route is through the internal
interface. So the packet should have been coming from the internal interface.

Although strict RPF is more secure, it can backfire if you use dynamic routing.
Dynamic routes can change quickly, and this fact combined with strict reverse path
forwarding could cause FortiGate to drop packets each time the preferred route
changes.
DO NOT REPRINT  Routing

© FORTINET

Some dynamic routing protocols require access to an interface that is always up.

A loopback interface isn’t correlated to any of FortiGate’s physical links. It exists in

the FortiGate software only. So all traffic with that destination stops at your
FortiGate. A loopback interface is always up and available, regardless of physical
cabling.

To create a loopback interface, go to System > Network > Interface and click on
Create New. The type must be loopback Interface.
DO NOT REPRINT  Routing

© FORTINET

Link aggregation is when multiple physical interfaces are logically bound into a
single channel. This increase bandwidth and provides redundancy between two
network devices.
DO NOT REPRINT  Routing

© FORTINET

WAN link load balancing, on the hand, consists of a group of interfaces connected
to multiple ISPs. Once created, the FortiGate sees all those Internet interfaces as
one single logical interface, the virtual WAN link. This helps to simplify the
configuration as now the administrator only needs to configure a single set of routes
and firewall policies that will be applied to all the ISPs.

There can be only one virtual WAN link per VDOM.

DO NOT REPRINT  Routing

© FORTINET

How FortiGate distributes traffic across its WAN links is very similar to how ECMP
does it. It can be based on:
• source IP address,
• source and destination IP addresses,
• interface’s weight, or
• spillover (like ECMP)
However, in WAN link load balancing, there is one more method, called “measured
volume.” With this method, sessions are distributed among all the links based on
each link current bandwidth utilization.
DO NOT REPRINT  Routing

© FORTINET

To configure WAN link load balancing, you need to specify which interfaces are
going to be members. In other words, which interfaces are connected to the
Internet. For each member, you can configure health check. If the health check fails,
the member is removed from the WAN link load balancing.
DO NOT REPRINT  Routing

© FORTINET

Optionally, you can be more selective and specify that specific traffic services are
routed through specific interfaces that are members of the virtual WAN. Additionally,
you can configure the FortiGate to measure the quality of each link (by measuring
either the latency or the jitter). So, selected traffic services can then be routed to the
interface with the highest or lowest measured quality.
DO NOT REPRINT  Routing

© FORTINET

After WAN link load balancing have been configured, a logical interface with the
name wan-load-balance is automatically added to the FortiGate. What you need to
do next is to create the routes and firewall policies that are going to be applied to all
the members of the virtual WAN.
DO NOT REPRINT  Routing

© FORTINET

Common routes are used to build a path so that the source can reach the
destination. Black hole routes do the opposite, making the destination unreachable.

Sometimes administrators require the use of wide summarized subnets. To avoid

unnecessary traffic, packets to unused subnets must be dropped. To do this, you
can create a black hole route to silently drop unwanted traffic.

In the above example, all spoke sites (R3, R4 etc.) use addresses in the
172.16.0.0/16 range. They have a routing protocol within their domain to reach the
specific 172.16.x.0/24 subnets. They also have a default route to access the
internet. The link between R1 and R2 is static only.

A packet sent from R3 whose destination is in the 172.16.0.0/16 range (but to a /24
network that does not exist) will take the default route path. R2 will then forward to
R1 and R1 will bounce this back to R2 because of the summarized static route. This
will continue until the packet TTL drops to 0. To prevent it, R2 should have a black
hole route for the network 172.16.0.0/16. In this way, if a packet is destined to a
subnet 172.16.x.0/24 that does not exist, it will be dropped and not forwarded to the
default route path (R1).
DO NOT REPRINT  Routing

© FORTINET

Multicast is traffic sent from one source to multiple destinations. A multicast routing
protocol populates the routing tables with information about how to route multicast
traffic.

Multicast is commonly used for video conferencing because it lowers the origin’s
resource usage and hardware requirements of transmitting to multiple destinations.
One stream of data goes to the router, which then multiplies that into data streams
for each destination.

A FortiGate device can be configure to route and apply NAT over multicast traffic.
DO NOT REPRINT  Routing

© FORTINET

We’ve seen the routing table in the GUI. Now, let’s see some diagnostics you can
use in the CLI.
This is the equivalent CLI command, which shows the routing table.

At the top, each code is defined.

Each route begins with a flag that shows what kind of route it is, or how it was
learned.

After the flag there is the route itself, then the distance and metric. Next you have
the gateway (if there is one), and the egress network interface. Finally, for dynamic
routes, you have a timer that indicates when the route will expire (if not renewed).
DO NOT REPRINT  Routing

© FORTINET

This command is very low-level. It shows the actual Forward Information Database
(FIB), which is the routing information that the kernel uses to route traffic.
DO NOT REPRINT  Routing

© FORTINET

This command gives a quick list of IP addresses associated with each interface.

They can be physical, VLAN, or virtual interfaces.

DO NOT REPRINT  Routing

© FORTINET

If you suspect that there is an IP address conflict, or that an IP has been assigned
to the wrong interface, you may need to look at the ARP table. This command is
used for that purpose. It shows the interface, IP address, and associated MAC
address.
DO NOT REPRINT  Routing

© FORTINET

The GUI offers a monitor to check the status of all the members of the virtual WAN
interface. It also shows the status of all the link health monitors configured in the
FortiGate.
DO NOT REPRINT  Routing

© FORTINET

To review, here is what we discussed. We talked about not only routing concepts
and configuration, but also diagnostics.