Sniffing network traffic in Python

Jose Nazario, Ph.D. <jose@monkey.org>

 Why Python?
Interpreted language
Bound to be slower than C

Rapid development
Easy data structure use
Fewer LoC per tool
Easy to manipulate strings
http://www.python.org/
 Marrying Python and Sniffing
Librares in C
Often SWIGged, exported to Python
pcap, dnet, nids
Modules
pypcap/pcappy pcap for python
dpkt packet deconstruction library
libdnet packet construction library (has python
bindings in the distribution)

pynids connection reassembly tool

libnids reassemble IP streams

NIDS E box (event generation box)

Userland TCP/IP stack
Based on Linux 2.0.36 IP stack
Uses libpcap, libnet internally
IP fragment reassembly
Userland

Kernel

IP stack
Userland

Kernel

Libnids IP stack
IP stack
 libnids Basics
Initialize
nids_init()
Register callbacks
nids_register_tcp()
nids_regster_ip()
nids_regiser_udp()
Run!
nids_run()
React
nids_kill_tcp()
 nids_run()

TCP callback UDP callback IP callback

TCP stream object: UDP packet: IP packet

- TCP state - source IP, port - struct IP packet
- client data - dest IP, port - contains upper
- server data - UDP payload layers
- source IP, port
- dest IP, port
- seq, ack, etc
 libnids TCP states
NIDS_JUST_ESTABLISHED
New TCP connected state (3WHS)
Must set stream->{client,server}.collect=1
to get stream payload collected
NIDS_DATA
Data within a known, established TCP connection
NIDS_RESET, NIDS_CLOSE,
NIDS_TIMED_OUT
TCP connection is reset, closed gracefully, or was lost

libnids doesnt expose SYN_SENT, FIN_WAIT, etc

 pynids Basics
Event driven interface (nids_run(),
nids_next())
TCP stream reassembly
TCP state exposure
Creates a TCP object
Holds addresses, data, etc
UDP and IP packet reassembly
 Basic pynids Steps
Initialize
nids_init()
Establish parameters
nids.param(attribute, value)
Register callbacks
nids.register_tcp(handleTcp)
def handleTcp(tcp):
Go!
nids_run()
while 1: nids_next()
 pynids Order of Operations
Packets come in

TCP?
State exist? Create state or reuse state
Append data
Process based on state in callback
UDP or IP?
Use handler, pass packet in
You process in callback
 Code Example (Python)
import nids
<handleTcpStream>

def main():
nids.param("scan_num_hosts", 0)
if not nids.init():
print "error -", nids.errbuf()
sys.exit(1)
nids.register_tcp(handleTcpStream)
try: nids.run() # loop forever
except KeyboardInterrupt:
sys.exit(1)
 Code Example (Python) cont
def handleTcpStream(tcp):
if tcp.nids_state == nids.NIDS_JUST_EST:
if dport in (80, 8000, 8080):
tcp.client.collect = 1
tcp.server.collect = 1
elif tcp.nids_state == nids.NIDS_DATA:
tcp.discard(0)
elif tcp.nids_state in end_states:
print "addr:", tcp.addr
# may be binary
print "To server:, tcp.server.data
print "To client:, tcp.client.data
 Code Example (C)
int main(int argv, char *argv[])
{
if (nids_init() == 0)
err(1, error, %s, nids_errbuf);
nids_register_tcp(handleTcp);
nids_run();
exit(0);
}
 Code Example (C), cont
int handleTcp(struct tcp_stream *tcp)
{
switch (tcp->nids_state) {
case NIDS_JUST_EST:
if ((tcp->addr.dest == 80) ||
(tcp->addr.dest == 8000) ||
(tcp->addr.dest == 8080) {
tcp.server.collect = 1;
tcp.client.collect = 1;
}
break;
case NIDS_DATA:
nids_discard(tcp, 0);
break;
case NIDS_CLOSE:
case NIDS_RESET:
case NIDS_TIMED_OUT:
printf(((%s, %d), (%s, %d))\n, inet_ntoa(tcp->saddr), tcp.srce,
inet_ntoa(tcp->daddr), tcp.dest);
printf(%s\n, tcp->server.data);
printf(%s\n, tcp->client.data);
break;
}
} About the same LoC, until we start string manipulation
 VersionDetect
Small python tool
Reports on headers
Fully passive
Support for: SSH (client, server), WWW
(client, server), and SMTP clients
Motivation: coordinate data collection with
TCP stack fingerprinting
63.236.16.161 SymbianOS 6048 (on Nokia 7650?) www 80/tcp
63.236.16.161: 80: Microsoft-IIS/6.0
 VersionDetect Output
192.168.1.7: 22: SSH-2.0-OpenSSH_3.5
192.168.1.101:http: Mozilla/5.0 (X11; U; OpenBSD i386; en-
US; rv:1.5a) Gecko/20031030 Mozilla Firebird/0.6.1
168.75.65.85: 80: Microsoft-IIS/5.0
165.1.76.60: 80: Netscape-Enterprise/3.6 SP2
168.75.65.69: 80: Microsoft-IIS/5.0
168.75.65.87: 80: Microsoft-IIS/5.0
69.28.159.7: 80: ZEDO 3G
198.65.148.234: 80: Apache/1.3.29 (Unix) PHP/4.3.3
216.150.209.231: 80: Apache/1.3.31 (Unix)
212.187.153.30: 80: Apache/1.3.31 (Unix)
212.187.153.37: 80: Apache/1.3.31 (Unix)
212.187.153.32: 80: thttpd/2.25b 29dec2003
64.209.232.207: 80: Apache/1.3.27 (Unix)
mod_perl/1.27
216.239.39.99: 80: CAFE/1.0
 http-graph
Small, passive python tool
Examines HTTP request header:
GET /blog/styles-site.css HTTP/1.1
Host: www.jackcheng.com
User-Agent: Mozilla/5.0 (X11; U; OpenBSD i386; en-US;
rv:1.5a) Gecko/20031030 Mozilla Firebird/0.6.1Accept:
text/css,*/*;q=0.1
Referer:
http://www.jackcheng.com/blog/archives/2004/12/ipod_rumo
rs.html
 http-graph
Directed graph history of browsing
Reconstructs graph from referrer and URL
in the header:
Referrer Request

Lets you view your history as you took it

Shows natural hubs of information
See also:
http://www.uiweb.com.nyud.net:8090/issues/issue37.htm
 Displaying http-graph Output
Writes a small dot file
dot part of graphviz tool
Use neato to graph
Output formats: SVG, PS, PDF, image map
Can make fully interactive!
Example http-graph Output
 Grabbing Data with pynids
tcp.{server, client}.data and just strings
Any string operations will work
Searching
if HTTP/1.0 in tcp.client.data:

Regular Expression searches

if re.search(HTTP/1.[10], tcp.client.data):

Rewriting
string.replace(req, GET HTTP/1.0, , 1)
 More Fun!
Privacy invasion
Snarf mail
Log conversations
IRC, AIM, etc
Steal files
FTP, P2P apps, HTTP downloads
Disrupt sessions
tcp.kill()

New dsniff is written in Python

 flowgrep
Marries sniffing with regular expressions
A lot like ngrep, tcpkill, and dsniff
Logs the whole connection, not just a packet
Look for data in streams using regular
expressions
Log or kill selected streams
Dirt cheap IDS or IPS
Under 400 lines of code
 Resources
http://www.tcpdump.org/
http://www.packetfactory.net/projects/libnids/
http://monkey.org/~provos/libevent/
http://monkey.org/~dugsong/{dpkt, pycap}
http://oss.coresecurity.com/projects/pcapy.html
http://monkey.org/~jose/software/flowgrep/
http://pilcrow.madison.wi.us/pynids/
 Additional Resources
Stevens, TCP/IP Illustrated vols 1 and 2
Schiffman, Building Open Source Network
Security Tools
RFCs from the IETF