Network Troubleshooting Tools

Kent Reuber ITS Networking Reuber@stanford.edu April 6, 2 !

"utline
#hat proble$s do %ou need to sol&e' Tool des(riptions )*A ti$e Tool des(riptions are in the +Software, se(tion of the -NA .uide/ http/00lnaguide0software.ht$l

#hat are the proble$s'

Are hosts online' 1ping2 3ow do %ou get to hosts' 1tra(eroute2 #hat are hosts running' 1n$ap2 #here0when ha&e hosts been seen' 1ip$2 +The network is slow, 1Netspeed, iperf2 4356 and 4NS 1S7Net reports2 #ireless proble$s 1&arious2 6a(ket sniffing 1wireshark2, and bat(h Net48 (hanges 1Net48 5-I2

6ing and tra(eroute

6ing/ Are %ou there'

6ing sends I596 e(ho re:uests to a host and asks for a repl%. Repl% ti$e is also returned. So$e hosts $a% (hoose not to repl% b% se(urit% poli(%. It $a% not $ean that the%;re down. Stanford de<prioriti=es pings at so$e of our borders, so a long ping ti$e or dropped pings does not indi(ate a poor (onne(tion. Stanford $aintains a spe(ial host/
+ping<$e.stanford.edu, >?e$pt fro$ ping filter. 3a&e outside users ping +ping<$e, if the% (lai$ that (onne(tions to Stanford are una&ailable or slow.

6ing for Ad&an(ed 7sers

5an in(rease pa(ket si=e to see duple? errors. 17ni?/ ping <s2
4efault s$all 1@6 b%te2 ping pa(kets don;t generate enough traffi( to show duple? proble$s. Tr% using pings of A B b%tes.

7se n$ap or si$ilar utilit% for +ping sweeps, of entire networks/

+n$ap <s6 @network rangeC, 1>?. +n$ap <s6 A!A.6D.AE. 02F,2 N$ap/ http/00inse(ure.org0

Tra(eroute/ 3ow do I get there'

3ow tra(eroute works/
Sour(e sends a series of pa(kets with in(reasing ti$e<to<li&es. 1TT- is the allowed nu$ber of router hops.2 7ni?09a(/ 746, #indows +tra(ert,/ I596. Routers will de(re$ent TT- and respond with an I596 +unrea(hable, $essage if TT- is . -ike ping, a ti$esta$p is returned.

QuickTime and a TIFF (LZW) decompressor are needed to see this picture.

Tra(eroute notes
Routers need not repl% to tra(eroutes. -a(k of a repl% does not $ean that the router is down. Return traffi( doesn;t ne(essaril% use the sa$e path.
This (an (ause proble$s with firewalls and pa(ket shapers that assu$e the% see the whole (on&ersation. #hen troubleshooting (onne(tion proble$s, %ou $a% want to ha&e the destination send tra(eroutes to %ou as well.

n$ap

N$ap/ S(anning nets

In addition to ping s(ans, %ou (an s(an for open ports on hosts. This (an be useful for seeing who is running a ser&i(e 1intentionall% or otherwiseG2 9% re(ipe for s(anning for open T56 ports/
,n$ap <6 <sT net <p ports <o. < H grep open,

.etting n$ap
4ownload fro$ http/00inse(ure.org 7ni? and 9a("S I usuall% re:uire (o$piling fro$ sour(e. #indows binar% a&ailable.

ip$

I69/ I6 @<C 9A5 addresses

Stanford<spe(ifi( utilit% 3ow it works/
4e&i(es broad(ast AR6 pa(kets when the% need to (o$$uni(ate lo(all%. Routers see these AR6 and (a(he it. Infor$ation is periodi(all% har&ested and kept in a database. 7sing I69, %ou (an tra(k when an I609A5 was first and last seen and where.

I69/ #hat;s it good for'

Jou (an find 9A5 addresses whi(h aren;t in Netdb. Kind out where a parti(ular de&i(e has been seen. See if $ultiple de&i(es are using a single I6 address.

9ore on I69
#here is it/
AKS/ 0usr0pubsw0sbin0ip$ Note/ this dire(tor% is not in %our default 6AT3. #ild(ards/ +L, 1single (hara(ter2, +M, 1$ultiple (hara(ters2 Run +ip$ <h, to see list of options.

7sing I69/

9A5 &endor (odes

9A5 addresses are DE<bit 16 b%tes2 ??/??/??/??/??/??, where ea(h +?, is a he?ade(i$al nu$ber <N,a<f. Kirst F b%tes are the "rgani=ationall% 7ni:ue Identifier 1"7I2, whi(h tell %ou who $ade the network (ard. 5an look this up. 9% fa&orite site/ http/00www.(offer.(o$0$a(Lfind0 5an tell %ou when Net48 re(ords are outdated. Kor e?a$ple, a Net48 re(ord for a 9a(intosh with 9A5 address / b/db 14ell2 is (learl% wrong.

Netspeed and Iperf

Netspeed * Iperf/ Speed testing

"ften hear +the network is slow,. 7seful tools/
Is it the (lient, the network or a ser&er' #here;s the bottlene(k' Netspeed 1#eb based speed to (a$pus ba(kbone2. Iperf 1(o$$and line tool for point<to<point2.

Netspeed
#eb based speed testing to Stanford ba(kbone/ http/00netspeed.stanford.edu0 or http/00iperf.stanford.edu0 7seful for finding duple? errors 1$is(onfigured hubs or swit(hes2 in the path.

Iperf
5o$$and line testing tool.
5an also run speed tests against netspeed.stanford.edu and iperf.stanford.edu 5an be run in ser&er $ode for testing speed between arbitrar% points 1e.g., within %our network2 http/00dast.nlanr.net06roOe(ts0Iperf0

3ow fast (an %ou go'

4S-/ A 9bps 1as%$$etri(2 E 2.AAb wireless/ A<P 9bps E 2.AAg wireless/ A<A2 9bps Kast >thernet/ E B 9bps .igabit/ '' Note/ (onsider these tests as upper bounds. Kor gigabit espe(iall%, %ou $a% not be able to transfer real data this fast.

4356

Troubleshooting 4356
9an% things (an go wrong. 6roble$s are rarel% (aused b% 4356 ser&er una&ailabilit%. Things to (he(k/
#hat I6 is the host getting' Netdb re(ord for the host. 4356 ser&er logs, roa$ing pool utili=ation reports.

7nderstanding 4356
Stanford has two 4356 ser&ers/ dusk and dawn. Info fro$ Netdb is uploaded appro?i$atel% e&er% AP $inutes. .i&e Netdb the ti$e to upload data. At Stanford, 9A5 address infor$ation is re:uired for su((essful 4356. Initial 4356 is a four step pro(ess using broad(astsQ renews are different.

QuickTime and a TIFF (LZW) decompressor are needed to see this picture.

-eases
4356 addresses are &alid for a li$ited period 1wired and wireless2. 3osts will re<(onfir$ their leases halfwa% through the lease period.
5lients use uni(ast dire(tl% to the 4356 ser&er 1(lients ha&e an address and the% know who their ser&er is2. Renew $essage t%pe is used. Nor$al 4356/ 2 da%s Roa$ing 4356/ D2 $inutes

4356 roa$ing
If the Netdb re(ord has a +ho$e, I6 address appropriate for the network where the de&i(e is lo(ated, 4356 ser&ers will send it.
5an ha&e +ho$e, I6 addresses and still be able to roa$ to other networks. 5an ha&e $ultiple +ho$e, addresses bound to ea(h 9A5 address.

If no appropriate address is entered, 4356 will look for a&ailable roa$ing addresses on the lo(al network.
Nu$ber of roa$ing address is spe(ified b% the -NA. 4efined in the Netdb network re(ord. 7suall% there are onl% a handful of roa$ing addresses. 5an easil% run out of the$.

#hat address did %ou get'

The address re(ei&ed $a% tell %ou what the proble$ is. Self assigned 1A6N.2PD.R.R2/
Net48 re(ord not set up properl%. No roa$ing address a&ailable. Routing or 4356 ser&er proble$ 1less likel%2.

A .?.?.?/

AN2.A6E.R.R/

7sed b% Network self<registration s%ste$. 1SNSR2 5ould also be used b% a rogue. 6robabl% a rogue 4356 ser&er.

Kinding rogues
Tr% pinging the gatewa% that;s being distributed. 7se +arp, (o$$and to get the 9A5 address of the gatewa%. "r use a sniffer if %ou ha&e one. -ook at swit(h 9A5 tables and find the offending hosts. Shut off the port or go ha&e a +(hat,. New Net<to<Swit(h (onfigs blo(k rogue 4356 ser&ersG

A&ailable 4356 reports

4536 logs for a gi&en host.
T%pe in 9A5 address and see the (on&ersation. Takes pra(ti(e to read. 3ow $an% roa$ing addresses were used in a da%. 3ourl% logs show nu$ber of 4356 $essages for hosts. +No free leases, $a% indi(ate that %ou;re out of roa$ing addresses.

Roa$ing address utili=ation

4356 reports fro$ dusk and dawn

All reports are linked fro$ -NA .uide software se(tion/ http/00lnaguide0software.ht$l

4NS

4NS at Stanford
3ost infor$ation is entered in Net48
7ploads to 4356 ser&ers about e&er% AP $inutes. 7ploads to 4NS ser&ers about e&er% hour.

Starts at P $inutes after the hour. Takes about 2 $inutes. Should be done b% F $inutes past the hour. Spe(ifi( info on ti$ing is kept in the Net48 help files.

4NS inspe(tion tools

Standard/ +host,, +nslookup,, +dig,. Stanford whois (an show %ou $ost Net48 infor$ation/
+whois <h whois.stanford.edu @:uer%C, 7se +M, and +L, as wild(ards as per ip$. .reat for people who need +read<onl%, a((ess, sin(e %ou don;t need a Net48 a((ount. Kor host na$es, %ou need to end :uer% in a +., or spe(if% +.stanford.edu, so that whois knows %ou want infor$ation on a host.

#ireless

#ireless proble$s
#ireless is slow or una&ailable. Reports (an be &ague. +#ireless is slow on the 2nd floor., Isolating the proble$ (an speed resolution.
>?a(tl% where is the proble$ o((urring' #hat a((ess point is the user (onne(ting to' 4o others ha&e proble$ in the area'

#ireless tools
A((ess point asso(iation/
9a(/ Internet 5onne(t utilit% 65/ ''

A((ess point dis(o&er% for seeing a&ailable A6;s and (hannels/ NetStu$bler, iStu$bler Iperf and Netspeed are useful for (he(king speed proble$s. "ften, a A6 reboot will sol&e the proble$.
A6 Oa(k 1tso2 infor$ation is in Netdb. 5an unplug and replug if ne(essar%.

6a(ket sniffer

>ther6eek and #ireshark

Stanford has site li(ense for >therpeek, but it;s still e?pensi&e. #ireshark 1for$erl% >thereal2 is free. 19otto/ +Sniff free or dieG,2
I windows appli(ation for 7ni?09a(. 8inar% for #indows. http/00www.wireshark.org0 So$e books are a&ailableG

Ad&i(e on Sniffing
Need for a sniffer is rare, but in&aluable when %ou need it. Jou will need to set up spe(ial +span, ports on %our swit(hes to see all traffi(.
No need if %ou;re interested in broad(asts and $ulti(asts. 9ost useful for seeing traffi( entering and lea&ing %our net. -earn to use it before %ou need itG

Net48 5o$$and -ine

Net48 5-I o&er&iew

4esigned for power users. 6ro&ides a subset of Net48 fun(tionalit% 1$ostl% nodes2 for bat(h (hanges. New features are periodi(all% added. 7se with (aution. Tr% one or two hosts before doing big bat(hes.

3ow to run Net48 5-I

-o(ated in AKS spa(e/
0usr0pubsw0sbin0netdb 1note/ this dire(tor% is probabl% not in %our 6AT32 7se <h option to get (o$$and s%nta?

Stuff %ou (an do 1to a single $a(hine or list of $a(hines2/

5hange ad$inistrators, lo(ations. 5hange I6 addresses. 4elete nodes.

)*A''