White Paper April 2005

McAfee System Protection Solutions

Buffer Overflow
Exploits: The Why and How

www.mcafee.com
 Buffer Overflow
Exploits: The Why and How 2

Table of Contents

Unchecked buffers 3
Exploiting the overflow 3
Now, for the clever part 3
Other types of buffer overflow exploits 4
Preventing buffer overflow exploits 4
Summary 4

www.mcafee.com
 Buffer Overflow
Exploits: The Why and How
Buffer overflow exploits are the tool of choice of
today’s attacker. These exploits have the most
power, are the easiest to use, and are all too
common. Buffer overflows constitute the largest
single threat to enterprises today.
• Buffer overflow exploits are very common. There
are hundreds of known unchecked buffers that
can be overflowed by hackers with more being
discovered all the time. Over 60% of CERT
advisories deal with buffer overflow exploits.
• Buffer overflow exploits are easy to use. Nearly
anyone (12 year olds and script kiddies
included) can download buffer overflow attack
code and follow a simple “recipe” to execute it.
No advanced technical knowledge is necessary
to run pre-written buffer overflow exploit code.
• Buffer overflow exploits are very powerful. In
many cases, the malicious code that executes
as a result of a buffer overflow will run with
administrator-level privileges, and therefore can
do anything it wants to the server.
Unchecked buffers
Unchecked buffers enable buffer overflows.
Unchecked buffers occur when the programmers
who write an application neglect to check data size.
In most commonly-used programming languages,
buffers to store data are allocated with a fixed size.
Programmers allocate enough buffer space to store
“normal” data. Let’s take an example:
Assume for a moment that you are a programmer in
charge of writing a login routine for an application. Your
routine must accept a user name and password from the
user and determine whether or not to allow the user
access. One of the first things you must determine is how
much buffer space to allocate for the user name and
password buffers. Given that the maximum length of any
user name for this system is 32 characters, you decide to
allocate 100 characters of buffer memory, which is more
than enough to hold the longest username.
You may erroneously think that since the 100 character
buffer is much larger than the 32 characters that a user
would normally submit, this routine is not vulnerable to
buffer overflow exploits. However, because you neglected
to check the length of the data submitted before copying it
into the buffer, this routine can be exploited. An attacker
only needs to send more than 100 characters to this
routine, and the buffer will overflow.
Exploiting the overflow
In the simplest case, an overflowed buffer will cause
the application that owns that buffer to crash, which
www.mcafee.com
3
results in denial-of-service. However, in most cases,
attackers can do much more. By cleverly
constructing the data submitted, attackers can cause
their arbitrary attack code to be executed. This
attack code can do nearly anything the attacker
desires, from reformatting the hard drive, to stealing
data, installing backdoor programs, etc.
Computer programs are organized into subroutines.
The program’s main routine calls each subroutine,
which performs its particular function(s) and then
returns control to the main routine.
Each subroutine, in turn, has to save various pieces
of information in order to perform its work.
Subroutines use an area of memory called the stack
for storing this information. One of these pieces of
information is the memory address to which the
subroutine should return control when it is finished
with its work.
Subroutines also store temporary data (buffers) on
the stack. Each time a subroutine is run, the
required memory is allocated on the stack in a unit
called a stack frame. This stack frame includes
space for any buffers the subroutine requires, as
well as the calling routine’s return address. When
the subroutine completes its work, it returns control
to the calling routine by jumping to the address
stored in the stack frame, and the stack frame is
deleted.
All of this machinery works perfectly until a stack
buffer overflows. When a user sends 1000
characters to a 100 character stack buffer, the extra
900 characters overwrite adjacent memory in the
stack frame, overwriting other buffers and the stack
frame’s return address.
Now, when the subroutine attempts to return control
to the main program, it jumps to the address that is
stored in the return address portion of the stack
frame. Unfortunately, this address has been
overwritten by the overflowed buffer and the address
is corrupted. When the program tries to jump to a
nonexistent address, the program crashes.
Now, for the clever part
If the attacker sends 1000 characters that are
carefully chosen, he or she can control the return
address. Rather than jumping to a non-existent
address, the attacker can instruct the program to
jump to the address of malicious exploit code. There
are two tasks that must be performed to accomplish
 Buffer Overflow
Exploits: The Why and How 4

this: new avenues of buffer overflow attack that are more

1. Loading the malicious code and more difficult to detect and prevent.
2. Executing the malicious code
Preventing buffer overflow exploits
Overflowing a stack buffer achieves both of these
tasks. The attacker sends a very long string of input Buffer overflow exploits can be prevented. If
data to the program. The input data includes the programmers were perfect, there would be no
malicious code as well as the address of that code. unchecked buffers, and consequently, no buffer
When this input data overflows the stack, the overflow exploits. However, programmers are not
malicious code is loaded into stack memory, and the perfect, and unchecked buffers continue to abound.
subroutine’s return address is overwritten to point to When unchecked buffers are found, vendors often
that malicious code. When the subroutine release patches that correct the problem.
terminates, the program jumps to the malicious Unfortunately, keeping patches up to date on a large
code, and that code is executed. The results of numbers of systems is difficult and many system
executing the attacker’s malicious code could be administrators fall behind in patch deployment.
catastrophic. If the malicious code reformats the
system’s hard drive, crucial data may be lost, and Another way to prevent buffer overflow exploits is to
significant time will be wasted rebuilding the system. use McAfee Entercept. Along with protecting the
Just as easily, the malicious code could attack other operating system, applications, and data, Entercept
machines, install backdoors, steal passwords, or any contains patented technology that blocks the
number of other possibilities. The malicious code execution of code from overflowed buffers. By doing
runs under the process context of the application it is this, Entercept prevents systems from being
attacking. Thus, if the application has root- or compromised by buffer overflow exploits. This
administrator-level privileges, the code will run as protection provides a layer of security that largely
root and be able to execute any command. If the eliminates the most significant threat to servers:
application is not running as root, the attacker can buffer overflow exploits. System administrators
still use a buffer overflow exploit to load a privilege should still patch their systems, but using Entercept
escalation exploit which would then give the desired buys the administrator crucial time to test and deploy
root privileges. Thus, buffer overflow exploits are patches while Entercept protects them from buffer
overflow threats. Entercept prevents all three major types
very useful to attackers.
of buffer overflow exploits: stack-based, heap-based, and
return-into-libc. It is the only security product on the
Other types of buffer overflow exploits market that can block all of these exploit types.
The stack-based overflow technique discussed here
is the most common type of buffer overflow exploit. Summary
However, other techniques are gaining popularity. Buffer overflow exploits are here to stay. They are
Heap-based overflows act similarly to stack-based pervasive, powerful, and easy to use. They are the
overflows, but overflow buffers on the heap. Return tool of choice to today’s attacker, and must be
into- libc exploits use buffer overflows on the stack prevented. Keeping systems up-to-date with the
or heap to cause execution of code in the system’s most current security patches and using McAfee
own libc library. These newer techniques provide Entercept will protect servers against these powerful
threats.

McAfee, Inc. 3965 Freedom Circle, Santa Clara, CA 95054

McAfee, and/or additional marks herein are registered trademarks or tradermarks of McAfee, Inc. and /or its affiliates in the U.S. and/or other countries. McAfee Red in
connection with security is distinctive of McAfee brand products. All other registered and unregulated trademarks herein are the sole property of their respective owners. ©
2005 McAfee, Inc. All Rights Reserved. 6-sps-ent-boe-003-0405

www.mcafee.com