Network

Troubleshooting
Tools
Outline
What problems do you need to
solve?
Tool descriptions
Q&A time

Tool descriptions are in the

Software section of the LNA
Guide:
http://lnaguide/software.html
What are the problems?
Are hosts online? (ping)
How do you get to hosts? (traceroute)
What are hosts running? (nmap)
Where/when have hosts been seen?
(ipm)
The network is slow (Netspeed,
iperf)
DHCP and DNS (SUNet reports)
Wireless problems (various)
Packet sniffing (wireshark), and
batch NetDB changes (NetDB CLI)
 Ping and
traceroute
Ping: Are you there?
Ping sends ICMP echo requests to a host and
asks for a reply. Reply time is also returned.
Some hosts may choose not to reply by security
policy. It may not mean that theyre down.
Stanford de-prioritizes pings at some of our
borders, so a long ping time or dropped pings
does not indicate a poor connection.
Stanford maintains a special host:
ping-me.stanford.edu
Exempt from ping filter.
Have outside users ping ping-me if they claim
that connections to Stanford are unavailable or
slow.
Ping for Advanced
Users
Can increase packet size to see
duplex errors. (Unix: ping -s)
Default small (<60 byte) ping
packets dont generate enough
traffic to show duplex problems.
Try using pings of 1000+ bytes.
Use nmap or similar utility for
ping sweeps of entire networks:
nmap -sP <network range> (Ex.
nmap -sP 171.64.18.0/23)
Nmap: http://insecure.org/
Traceroute: How do I
get there?
How traceroute works:
Source sends a series of packets with increasing
time-to-lives. (TTL is the allowed number of router
hops.) Unix/Mac: UDP, Windows tracert: ICMP.
Routers will decrement TTL and respond with an ICMP
unreachable message if TTL is 0.
Like ping, a timestamp is returned.

QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
Traceroute notes
Routers need not reply to
traceroutes. Lack of a reply does
not mean that the router is down.
Return traffic doesnt necessarily
use the same path.
This can cause problems with firewalls
and packet shapers that assume they see
the whole conversation.
When troubleshooting connection
problems, you may want to have the
destination send traceroutes to you as
well.
nmap
Nmap: Scanning nets
In addition to ping scans, you
can scan for open ports on
hosts.
This can be useful for seeing
who is running a service
(intentionally or otherwise!)
My recipe for scanning for open
TCP ports:
nmap -P0 -sT net -p ports -oG - |
grep open
Getting nmap
Download from
http://insecure.org
Unix and MacOS X usually
require compiling from
source.
Windows binary available.
ipm
IPM: IP <-> MAC
addresses
Stanford-specific utility
How it works:
Devices broadcast ARP packets when
they need to communicate locally.
Routers see these ARP and cache
it.
Information is periodically
harvested and kept in a database.
Using IPM, you can track when an
IP/MAC was first and last seen and
where.
IPM: Whats it good
for?
You can find MAC addresses
which arent in Netdb.
Find out where a particular
device has been seen.
See if multiple devices are
using a single IP address.
More on IPM
Where is it:
AFS: /usr/pubsw/sbin/ipm
Note: this directory is not in
your default PATH.
Using IPM:
Wildcards: _ (single
character), % (multiple
characters)
Run ipm -h to see list of
options.
MAC vendor codes
MAC addresses are 48-bit (6 bytes)
xx:xx:xx:xx:xx:xx, where each x is a
hexadecimal number 0-9,a-f.
First 3 bytes are the Organizationally
Unique Identifier (OUI), which tell
you who made the network card.
Can look this up. My favorite site:
http://www.coffer.com/mac_find/
Can tell you when NetDB records are
outdated. For example, a NetDB record
for a Macintosh with MAC address
00:0b:db (Dell) is clearly wrong.
Netspeed and Iperf
Netspeed & Iperf:
Speed testing
Often hear the network is
slow.
Is it the client, the network or
a server?
Wheres the bottleneck?
Useful tools:
Netspeed (Web based speed to
campus backbone).
Iperf (command line tool for
point-to-point).
Netspeed
Web based speed testing to
Stanford backbone:
http://netspeed.stanford.e
du/
or
http://iperf.stanford.edu/
Useful for finding duplex
errors (misconfigured hubs
or switches) in the path.
Iperf
Command line testing tool.
Can also run speed tests against
netspeed.stanford.edu and
iperf.stanford.edu
Can be run in server mode for
testing speed between arbitrary
points (e.g., within your
network)

http://dast.nlanr.net/Projects/Ip
erf/
How fast can you go?
DSL: 1 Mbps (asymmetric)
802.11b wireless: 1-5 Mbps
802.11g wireless: 1-12 Mbps
Fast Ethernet: 80+ Mbps
Gigabit: ??
Note: consider these tests as
upper bounds. For gigabit
especially, you may not be able
to transfer real data this
fast.
DHCP
Troubleshooting DHCP
Many things can go wrong.
Problems are rarely caused
by DHCP server
unavailability.
Things to check:
What IP is the host getting?
Netdb record for the host.
DHCP server logs, roaming pool
utilization reports.
Understanding DHCP
Stanford has two DHCP servers: dusk and dawn.
Info from Netdb is uploaded approximately every
15 minutes. Give Netdb the time to upload data.
At Stanford, MAC address information is required
for successful DHCP.
Initial DHCP is a four step process using
broadcasts; renews are different.

QuickTime and a
TIFF (LZW) decompressor
are needed to see this picture.
Leases
DHCP addresses are valid for a
limited period (wired and
wireless).
Normal DHCP: 2 days
Roaming DHCP: 42 minutes
Hosts will re-confirm their leases
halfway through the lease period.
Clients use unicast directly to the
DHCP server (clients have an address
and they know who their server is).
Renew message type is used.
DHCP roaming
If the Netdb record has a home IP address
appropriate for the network where the device
is located, DHCP servers will send it.
Can have home IP addresses and still be able
to roam to other networks.
Can have multiple home addresses bound to
each MAC address.
If no appropriate address is entered, DHCP
will look for available roaming addresses on
the local network.
Number of roaming address is specified by the
LNA. Defined in the Netdb network record.
Usually there are only a handful of roaming
addresses. Can easily run out of them.
What address did you
get?
The address received may tell you what
the problem is.
Self assigned (169.254.*.*):
NetDB record not set up properly.
No roaming address available.
Routing or DHCP server problem (less
likely).
10.x.x.x:
Used by Network self-registration system.
(SNSR)
Could also be used by a rogue.
192.168.*.*:
Probably a rogue DHCP server.
Finding rogues
Try pinging the gateway thats
being distributed.
Use arp command to get the MAC
address of the gateway. Or use
a sniffer if you have one.
Look at switch MAC tables and
find the offending hosts. Shut
off the port or go have a
chat.
New Net-to-Switch configs block
rogue DHCP servers!
Available DHCP reports
DCHP logs for a given host.
Type in MAC address and see the conversation.
Takes practice to read.
Roaming address utilization
How many roaming addresses were used in a day.
DHCP reports from dusk and dawn
Hourly logs show number of DHCP messages for
hosts.
No free leases may indicate that youre out
of roaming addresses.
All reports are linked from LNA Guide
software section:
http://lnaguide/software.html
DNS
DNS at Stanford
Host information is entered in
NetDB
Uploads to DHCP servers about
every 15 minutes.
Uploads to DNS servers about
every hour.
Starts at 5 minutes after the hour.
Takes about 20 minutes. Should be
done by 30 minutes past the hour.
Specific info on timing is kept in
the NetDB help files.
DNS inspection tools
Standard: host, nslookup, dig.
Stanford whois can show you most
NetDB information:
whois -h whois.stanford.edu <query>
Use % and _ as wildcards as per ipm.
Great for people who need read-only
access, since you dont need a NetDB
account.
For host names, you need to end query in a
. or specify .stanford.edu so that
whois knows you want information on a
host.
Wireless
Wireless problems
Wireless is slow or unavailable.
Reports can be vague. Wireless
is slow on the 2nd floor.
Isolating the problem can speed
resolution.
Exactly where is the problem
occurring?
What access point is the user
connecting to?
Do others have problem in the area?
Wireless tools
Access point association:
Mac: Internet Connect utility
PC: ??
Access point discovery for seeing
available APs and channels:
NetStumbler, iStumbler
Iperf and Netspeed are useful for
checking speed problems.
Often, a AP reboot will solve the
problem.
AP jack (tso) information is in Netdb.
Can unplug and replug if necessary.
Packet sniffer
EtherPeek and
Wireshark
Stanford has site license for
Etherpeek, but its still
expensive.
Wireshark (formerly Ethereal) is
free. (Motto: Sniff free or
die!)
X windows application for Unix/Mac.
Binary for Windows.
http://www.wireshark.org/
Some books are available!
Advice on Sniffing
Need for a sniffer is rare, but
invaluable when you need it.
Learn to use it before you need
it!
You will need to set up special
span ports on your switches
to see all traffic.
No need if youre interested in
broadcasts and multicasts.
Most useful for seeing traffic
entering and leaving your net.
NetDB Command Line
NetDB CLI overview
Designed for power users.
Provides a subset of NetDB
functionality (mostly
nodes) for batch changes.
New features are
periodically added.
Use with caution. Try one
or two hosts before doing
big batches.
How to run NetDB CLI
Located in AFS space:
/usr/pubsw/sbin/netdb (note: this
directory is probably not in your
PATH)
Use -h option to get command syntax
Stuff you can do (to a single
machine or list of machines):
Change administrators, locations.
Change IP addresses.
Delete nodes.
Q&A??