DO NOT REPRINT

FORTINET

Intrusion Prevention System

In this lesson, we will show you how to use FortiGate IPS. IPS is part of what
makes FortiGate a UTM that can keep pace with the latest attacks.
Beyond simply TCP stateful inspection and masking internal network IPs, modern
FortiGate UTM firewalls can detect and block exploit attempts in higher layer
protocols.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

After completing this lesson, you should have these practical skills. Essentially, you
will learn how to use your FortiGate to study what is normal for your network, then
detect and block rate anomalies and mechanism attacks.
Lab exercises can help you to test and reinforce your skills.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Before we begin, its important to understand: Not all attacks can be 100%
positively identified. Sometimes, there is uncertainty.
What is the difference between an attack and an anomaly? To compare, FortiGate
IPS uses attack signatures where it can detect an attack with relative certainty and
performance. But the IPS engine also can use heuristic methods to find statistical
anomalies unusual order in the packet flow, or suspicious volumes of certain
packet types. An example: the client uses the HTTP MKCOL method, but your web
site has only static web pages, so its suspicious to use a method for dynamic sites.
Many anomalies indicate a DoS attempt. So the IPS engine is also used by DoS
policies, except where its performed in specialized hardware FortiASIC
chips instead of in the kernel, on the CPU.
If an anomaly is actually normal for your specific network, to reduce false positives,
disable that signature in your IPS profile or DoS policy.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

Lets define what IPS currently means on FortiGate. You may be surprised.
On older systems, IPS might have meant purely a Snort-style signature matching. It
was similar to anti-virus signatures, but for protocols instead of files.
But on FortiGate UTM, IPS has evolved to also detect anomalous traffic patterns,
such as a flood of traffic exceeding the usual bandwidth volume, and to apply
heuristics that prevent an unexpected behavior of the protocol.
(click)
Why? Arent IPS signatures enough?
Some attacks cant be successfully or efficiently defined in a signature. If the attack
is qualitatively or quantitatively too similar to legitimate traffic, IPS false positives will
block your network service not the result you want.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

How does the IPS engine determine if a packet contains an attack or anomaly?
Protocol decoders parse each packet according to the protocol specifications. Some
protocol decoders do require a port number specification (configured in the CLI), but
usually, the protocol is automatically detected. If the traffic doesnt conform to
specification if, for example, it sends malformed or invalid commands to your
servers then the protocol decoder detects the error. For example, a stream of
packets might match the HTTP decoders pattern named
Cisco.CatOS.CiscoView.HTTP.Server.Buffer.Overflow.
(click)
A default, initial set is included in each FortiGate firmware. FortiGuard IPS service
updates them, sometimes daily, with new signatures. That way, IPS remain effective
against new exploits. Unless a protocol specification or RFC changes (which is not
very often), protocol decoders are rarely updated. The IPS engine itself changes
more frequently, but still not often.
What part of IPS is updated most? The IPS signatures. New signatures are
identified and built during the day by FortiGuard research teams, just like with antivirus. So if your FortiGuard Services contract expires, you can still use IPS.
However, just like with anti-virus scans, IPS scans will over time become
increasingly ineffective old signatures wont defend against new attacks.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Regular updates are vital. If your FortiGate doesnt have the latest signatures,
your network is vulnerable. Always make sure that your FortiGate has a reliable
Internet connection, and that it is scheduled to often request updates from
FortiGuard.
What is included in a FortiGuard IPS update? Protocol decoders, the engine, and
signatures. The signature database is subdivided into Regular and Extended.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Regular signatures are common attacks whose signatures, during testing prior to
release on the FortiGuard Distribution Network, caused rare or no false positives.
So its a smaller database, and its default action is to block the detected attack.
Extended signatures contain everything else. In FortiOS 5.2, the IPS extended
database is enabled by default for all FortiGate models that
have multiple CP8. Otherwise, they are disabled, because either:
Performance impact is significant, or
Nature of the attack doesnt support blocking
By default, the Regular database is selected, not the Extended. In fact, due to its
size, the extended database is not available for FortiGate models with a smaller
disk and/or RAM. But for high security networks, you may be required to enable
extended signatures. In that case, you should mark the Enable Extended IPS
Signature Package option on System > Config > FortiGuard.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

When your FortiGate downloads new IPS signatures, or a new engine, syntax may
change. So if you write your own custom signatures, especially after upgrading your
FortiGates firmware, you may need to check if its still compatible.
IPS involves anomaly inspection, deep packet inspection, full content inspection,
activity inspection, and heuristic detection. Some software does not maintain a
constant pattern. Skype and other peer-to-peer software, for example, periodically
change in order to avoid detection. So in order to correctly identify it, IPS requires
heuristics and adaptive detection.
As a result, FortiGuard IPS also provides updates for application control, for
example.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

When your FortiGate downloads a FortiGuard IPS package, new signatures will
appear in the signature list. For each sensor that uses a signature, when
configuring, you can change its Action setting.
The default often is correct, except if:
Your software vendor releases a security patch. Continuing to scan for exploits
will waste FortiGate resources.
Your network has a custom application with traffic that inadvertently triggers an
IPS signature. You can disable it until you notify Fortinet so that FortiGuard can
modify the signature to avoid false positives.
The list of IPS signatures also indicates the severity level. What do the indicators
mean?

DO NOT REPRINT
FORTINET

Intrusion Prevention System

The FortiGuard severity level is based on the CVSS 2 rating system. There are
many contributing factors. For details, go to the first.org web site.
Do all severity levels match CVSS exactly? No.
Fortinet always marks remote code execution as high or critical severity, regardless
of the CVSS rating. Details are explained on the FortiGuard web site.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Do you have the CVE ID or Microsoft ID for a specific vulnerability, but dont know if
there is a corresponding IPS signature yet?
On the FortiGuard web site, you can search for the latest IPS signatures. But you
can also read details about recently discovered zero-day attacks, white papers,
blogs and security advisories.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

If youre not sure if you should enable an IPS signature on your FortiGate, you can
search the FortiGuard web sites encyclopedia.
The encyclopedia has useful information such as affected systems and
recommended corrective actions. So if you dont use that protocol or dont have a
vulnerable system, you can safely disable the corresponding signature. But if you
are vulnerable, the encyclopedia can provide information about how to protect
yourself.
The FortiGuard encyclopedia only contains publicly disclosed vulnerabilities,
though. Obviously it cant contain vulnerabilities that, for whatever reason, cant yet
be responsibly disclosed.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Exploits for unknown vulnerabilities called zero-day attacks are sold for large
amounts of money on the black market. Since these exploits arent known to their
vendors, nor to security experts, theres no available patch or signature for
detection. Thats what makes them so dangerous.
Some companies and organizations like Facebook and Google have offered
bounties for the responsible disclosure of these exploits, but theres a very profitable
market for black hat hackers to sell these discoveries to everyone from covert
government surveillance to organized crime syndicates.
Zero-day attacks are the keys to your networks kingdom.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

If you notice an attack, your initial self-defense instinct may be to immediately take
the server offline, then format it to remove all traces of malware. But by doing this,
youll alert the attacker, and destroy forensic evidence. For motivated attackers, this
will only educate them their next attack will be harder to detect, and more
sophisticated. Make sure your PSIRT team understands the most appropriate
way to respond to each different type of intrusion.
If youre vigilant, and if you have the resources, you can also write your own custom
IPS signatures. Well talk about how to do that next.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Before you write custom IPS signatures, lets first explain how the IPS engine
works.
FortiGate doesnt compare traffic to each signature individually. This would require
the CPU to load from disk and then evaluate each complete signature. In total,
when fully enabled, this would be more than 8,000 disk accesses and comparisons.
So instead, IPS compiles them into a decision tree, similar to the example shown
here.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

FortiGate loads this entire decision tree into RAM. This can increase memory usage
significantly, especially on desktop FortiGate models that dont have much RAM. So
if your RAM usage is already high, you should reduce it first before enabling IPS.
Otherwise, your FortiGate may immediately enter conserve mode, and refuse to
accept any more configuration changes! But the advantage is that the tree takes
much less CPU and total RAM for a full IPS scan.
To make the tree, FortiGate breaks down signatures into identical pieces port,
protocol, etc. and shares the evaluation. So if traffic does not match that part, then
the IPS engine can bypass comparisons with all similar signatures. But if it does
match, then IPS continues with the next shared segment of the signature. When it
finds a match, FortiGate applies its corresponding action.
Remember discussing the difference between attacks and anomalies? Detecting
uncertain attacks can require even more ongoing analysis, and more RAM to store
traffic statistics. So if your CPU usage or RAM usage is high, and if you dont
require anomaly analysis for all protocols, clients, or servers, disable it. Better yet,
offload it to an NP FortiASIC if your FortiGate model has them. Hardware
accelerated anomaly detection can be configured in the CLI.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

To write custom signatures, first use packet capture to record packet samples.
Understand and avoid mismatches with normal packets on your network, including
at other OSI layers such as Layer 2 and Layer 3, which will be evaluated first.
Remember: if you misconfigure a custom signature, or if you configure a custom
signature that is no longer supported after you update the FortiGate firmware or IPS
engine, problems like this often arent included in Fortinet Technical Support. So if
possible, you should also test your custom signatures in a lab.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

Well show one example here.
(click)
All start with F-SBID(.
(click)
After that, protocol-specific key words define what part of the packet to search for a
match, and what values comprise a match. Usually, a keyword is followed by a
corresponding value that is its setting, except for a few standalone keywords such
as --no_case. Each key-value pair ends with a semi-colon and a space. You can
include multiple key-value pairs. The signature ends with the closing parenthesis.
A reference to syntax for custom IPS signatures is in the FortiGate Handbook.
Supported key words vary by the protocol decoders. For example, the SMTP
protocol supports the VRFY command, and so there is a protocol decoder flag for
it.. So if you create custom signatures, you should be sure to read the Release
Notes and new Handbook before upgrading, and (if possible) test the firmware
before installing it in a live traffic environment. Lets see some examples.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

Here is a sample custom signature called Ping.Death. It searches for ICMP traffic
that exceeds about 32 KB.
(click)
After you create and save the signature, FortiGate will automatically add an attack
ID. So dont include it when you enter the signature.
(click)
Next is a signature for HTTP.
It searches for the pattern POST in a very specific location inside the packet. In
normal HTTP POST requests, the method should be in this specific location. This
prevents IPS from scanning the entire HTTP payload, which could contain a web
page that accidentally matches, for example, due to the words POSTAL CODE.
Your signature should be specific, but not too specific extra comparisons reduce
performance.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Once you have created your custom signature, pair it with an action within an IPS
sensor. Then reference that IPS sensor in a firewall policy.
The steps are the same, by the way, regardless of whether you want to use custom
signatures or ones predefined by FortiGuard.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Heres an example of an IPS filter being created.

To include all signatures in the filter, weve marked ALL options. To include only a
few signatures in the filter, we would only mark one option. For example, if we only
marked the Client option, only 4 signatures would be included in the filter.
Each individual signature can have multiple tags, such as HTTP, Microsoft, IIS, and
TCP. The more specific you can make your filter, the less resources will be used to
scan your traffic, because its parts will seldom match and so the IPS engine will
quickly continue with the next comparison or scan.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

When the IPS engine compares traffic with the signatures in each filter, order
matters. The rules are similar to firewall policy matching: topmost filters are
evaluated first, and the first match applies. Subsequent filters are skipped.
So position most likely matching filters at the top of the list, unless they might cause
false positives. (Position those last, so that FortiGate will test them only if no
previous, more sure signature matches.) Avoid making too many filters, since this
will increase evaluations and CPU usage. Also avoid making very large signature
trees in each filter, which will increase RAM usage all unique pieces of the attack
pattern must be loaded into RAM. Strike a balance. If an attack can be prevented in
hardware (by NP FortiASIC chips, for example), or by another method (by
disallowing an unnecessary protocol at the firewall level, for example), do this first.
Then, for the remaining, craft careful IPS sensors to protect relevant vulnerabilities.
For rate-based signatures (previously called anomalies), you can choose how to
match: by source IP, destination IP, DHCP Client MAC, or DNS Domain Name.
Choose whichever will generate the least entries yet behave correctly. For Internetfacing policies, this is unfortunately one that requires IPS to analyze many clients
connections: Source IP. So enable only rate-based signatures for vulnerable
protocols you actually use. Then block malicious clients for extended periods. This
saves system resources and can discourage a repeat attack: FortiGate will not track
statistics for that client while it is temporarily blacklisted.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

To apply an IPS sensor, enable IPS and then select the sensor in a firewall policy.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

So far weve shown signatures that match illegal commands and invalid protocol
implementations. Those are easy to confirm as an attack.
What about attacks that function by exploiting asymmetric processing, or bandwidth
between clients and servers? There are many ways to make a Denial of Service
attack. Some denial of service (DoS) attacks, for example, exhaust limited serverside bandwidth or sockets. Unless you know what bandwidth is abnormal for your
network, you may not be able to confirm an attack.
(click)
The goal is to overwhelm the target to consume resources until it cant respond to
legitimate traffic. This can be done in various ways. High bandwidth usage is only
one type of DoS. Many sophisticated DoS such as Slowloris dont require high
bandwidth.
For high-bandwidth DoS, remember that although your FortiGate blocks traffic
floods, the flood is still consuming bandwidth up to the point of its external interface.
So your servers are protected from impact, but if the upstream network is not, so
your servers may still be effectively unavailable. Especially for distributed denial of
service attacks, you must work with your ISP to fully prevent high-bandwidth DoS.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

To block DoS attacks, apply a DoS policy on a FortiGate that is between attackers
and all resources that you want to protect.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

DoS protection exists for 4 protocols: TCP, UDP, ICMP and SCTP. Each one has 4
different types of anomaly detection.
A flood sensor detects a high volume of that particular protocol, or signal in the
protocol.
Sweep/Scan detects attempts to map which of a hosts ports respond and
therefore may be vulnerable.
Source signatures look for large volumes of traffic originating from a single IP.
Destination signatures looks for large volumes of traffic destined for a single IP.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

If you do not have an accurate baseline for your network, then when you implement
DoS for the first time, be careful not to completely block network services. To
prevent this, initially configure the DoS policy to log but not block. Using the logs,
you can analyze and determine normal and peak levels for each protocol. Then
adjust the thresholds to comfortably, but not loosely, allow the usual peaks.
Thresholds that are too high can allow your resources to be exhausted before the
DoS policies trigger. Thresholds that are too low will cause FortiGate to drop normal
traffic.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

Now we will take a look at some common types of DoS attacks. The first is called a
SYN flood.
In TCP, the client sends a SYN signal to initiate a connection. The server must
respond, then remember the start of the connection in RAM while it waits for the
client to acknowledge (or ACK). Until ACK, the connection is only half-formed,
so the attack wont show up in a connection table. Normal clients will quickly
ACK and begin to transmit data. But malicious clients continue quickly or slowly,
to avoid detection to send more SYN packets, half-opening more connections,
until the servers table is full. Then, the server cannot accept more. It begins to
ignore all new clients. Depending on the system, this attack can also damage
hardware.
(click)
To defend against this, FortiGate acts as a pseudo-proxy. It waits until the client has
finished connection build-up to form the back-end connection. If this doesnt
complete quickly, FortiGate begins to drop the attackers connection requests from
the table.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

Another type of anomaly is an ICMP sweep. ICMP is used during troubleshooting:
devices will respond with success or error messages. But attackers can use this to
probe the network for valid routes and responsive hosts.
(click)
This provide information about your network before the attacker crafts more serious
exploits.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

An individual DoS attack is a flood of traffic coming from a single address. It can
originate from the Internet or even from your internal network. Typically a single
device makes many connections or sessions, and possibly uses much bandwidth to
a single location.
(click)
All four protocols in the DoS profile (ICMP, TCP, UDP, SCTP) have an anomaly
sensor for the source. These are built to examine the traffic each IP is generating
and compare that to the threshold value.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

A variation of this is the DDOS, or Distributed Denial of Service attack. It has many
of the same characteristics. The main difference is that multiple devices are all
attacking at the same time. This could be 5, or maybe 50, or 500 or more devices
attacking together.
(click)
Remember earlier when we showed that despite FortiGate protecting the host, the
resource could still become unavailable if the bandwidth to the ISP was consumed?
Think about how these detections work. They do not trigger until the threshold is
reached. Lets say, for example, that the DoS sensor doesnt trigger until 5000
sessions occur within 1 second. These 5000 sessions are allowed: first come, first
served. So if multiple external devices are all generating connections to the same
destination, attackers which are creating connections the fastest, will be the ones
most likely to get the connections. Many of these DoS attacks can physically
damage systems, so the goal is to prevent that from happening and prevent this
kind of damage.
But how can you find the right threshold? You must know what normal traffic
thresholds are on your network in other words, the baseline.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Everything we have shown so far is inline scanning: traffic passes through FortiGate
from one interface to another. But you can also deploy FortiGate outside of the
direct path of packets, in a one-arm topology with a monitor-only mechanism. This
is also called sniffer mode because it detects but does not block.
To do this, connect FortiGate to a switchs SPAN or mirroring port. The switch will
send a duplicate of egressing packets to FortiGate, which FortiGate then scans.
Notice that because its scanning a copy not the original packet it cant modify or
block the original packet.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

When should you use one-arm IPS?

Historically, when IPS scanning was first invented, it was slow. Old IPS could
introduce high latency. So one-arm deployment was common, but IPS on an inline
firewall wasnt.
Now, hardware performance is much better. And one-arm has a significant
limitation: one-arm FortiGate cannot block traffic. Because its on a mirrored port on
the switch, not directly in between the attacker and your protected network,
FortiGate isnt placed to intervene. So today, most people use one-arm only during
testing or evaluation. Think of one-arm IPS as log-dont-block.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Before sniffer mode, the only way you could demonstrate a FortiGate without
changing IP addresses was to put it transparently inline with the traffic. This could
potentially disrupt the network if you didnt understand the Layer 2 topology. But
now, there is no risk.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Sniffer mode is enabled on a FortiGates physical interface, not a logical interface

such as a VLAN.
After you select One-Arm Sniffer on an interface, you can choose any security
profile that uses the IPS engine. For example, you can use an application control
profile if it is flow-based, since flow-based scans use the same engine as IPS. (Onearm DLP is also configurable, but via the CLI only.)
FortiGate wont allow you to choose proxy-based profiles that arent supported in
one-arm inspection.
Why arent all profiles/actions supported? Its not technically possible. This is due to
the nature of the topology and asynchronous scanning. To modify traffic or proxy
connections, FortiGate must be in line not out of band on a SPAN port and
stop the packet until it finishes scanning. That is, inspection must be in sync with the
connection. However, one-arm scans after the interface has already forwarded the
packet. Scanning and forwarding are out of sync. Since the packet has already
egressed, FortiGate cant proxy or block. Thats why its not possible to support all
features in this mode.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Now lets see some logs that are generated by IPS.

Anomalies and signature matches have different logs associated with them.
Since an anomalys name already gives information about the traffic and the attack,
such as protocol and source address, many details in the logs arent needed.
But you often will require information about which applications or operating systems
are vulnerable. You also need to know the action whether FortiGate blocked or
simply monitored (detected) the attack. If you configured FortiGate to only monitor,
you may need to forensically investigate the targeted host. This is where host-based
tripwires can be useful.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

IPS sensors are not the only way that IPS can generate logs, however. When DoS
policies generate logs, they are aggregated. When several incidents occur together,
this reduces the number of log messages.
In large attacks, the number of incidents can easily reach 100,000 in a few
seconds. Generating a log entry for every packet that matches would completely
utilize the CPU. So instead, FortiGate collapses incidents by periodically recording
only one message for all of them, and noting the number of incidents.
Here, the detection threshold was 50, and the total count is 75. So FortiGate
doesnt make 24 separate log entries (1 for each incident above 50). Its just one log
message.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

What commands can you use if IPS is dropping packets unexpectedly?

In the CLI, use diag ips anomaly list to show all hosts that are currently
being limited by DoS policies, and by what signature. If theres no matching traffic,
then it will not display any output.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Another available diagnostic command is diag autoupdate version. This

lists various IPS databases and engines that are installed on the FortiGate.
It also displays the results of the last update attempt. So it can be useful if you
suspect interruptions to FortiGuard connectivity.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Another command that can be used is troubleshoot the IPS is diag test app
ipsm.
For example, you could type diag test app ipsm 99.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

(slide contains animation)

What does the IPSEngine actually do?
Notice that if you run the diag test app ipsm 5 command, and if you have
any kind of flow-based inspection profile, the CPU usage of the IPSEngine process
drops dramatically, but doesnt reach 0.
This is because IPSEngine is responsible for all of the things weve shown in this
class: intrusion protection, DoS policies and protocol decoders. Its also responsible
for application control, flow-based policies for antivirus, web filtering, email filtering,
and DLP. So relatedly, its also responsible for session helpers.
(click)
Session helpers arent an inspection option; they are automatic. To stop them, you
must stop IPSEngine.

DO NOT REPRINT
FORTINET

Intrusion Prevention System

Here is a review of what we discussed. We showed:

The difference between a signature that matches a known attack, versus one that
matches a traffic pattern anomaly
How protocol decoders find anomalies, and how this is different than proxy-based
scans
Severity levels
How to configure IPS sensors, including ones with custom signatures
Denial of Service attacks, which are a type of anomaly
One-arm deployment, both its limitations and purpose
IPS logs
Diagnostic commands for IPS, including expected output, since some processes
of the IPS engine are used by other scans