24/04/2015

DocumentDisplay

IdentifyingafailedEthernetcableonanOracleDatabaseMachine(Exadata)(DocID
1994662.1)

APPLIESTO:
OracleExadataHardware
Informationinthisdocumentappliestoanyplatform.

PURPOSE
Thesymptomsofacablefailurearemany.SomeofthemoreobviousisthelinklightontheswitchortheNICwillnot
light.Anothermightbeadegradednetworkbandwidth.TimeDomainReflectometer(TDR)needstobeexecuted
fromtheswitchandcan'tbedonefromthehost.TDRrequiresspecificcircuitrythatisnottypicallyavailableonthe
networkinterfacecard.
TouseTDR,youwillneedenableprivilegesontheswitchtoissuethecommandsneeded.Ethtoolisanothertoolto
usedtovalidateTDR'soutput.EthtoolshowsthesettingsontheNICsoonecancomparethemtothesettingsonthe
switch.
FormoreinformationonCisco'simplementationofTDRsee
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12
2SX/configuration/guide/book/intrface.html#wpmkr1066782

TROUBLESHOOTINGSTEPS
Beforecoveringthecommandsused,let'sreviewtheconstructionofTwistedPaircablingtoproperlyunderstandthe
switchdiagnosticoutput.Category5orCategory6TwistedPairEthernetcableconsistsoffourpairsoftwistedcopper
wirethatconnectinaspecificwayintoanRJ45plug.TheTDRdiagnosticrunstestsoneachtwistedpairto
determinehowlongtheoverallcableis,orhowfarfromtheswitchapotentialbreakislocated.Thediagnosticsalso
liststhenetworkspeedoftheNIC.Duringthetest,connectivitybetweentheswitchandhostisterminatedwhilethe
testisbeingrun,soitisbesttorunthetestduringasystemoutageperiod.
NOTE:TheinformationinthisnotewasacquiredusingaCiscoCatalyst4948switchrunningVersion15.0(2)SG8of
theCIOS.ItshouldbevalidforotherCiscoequipmentthathasTDRavailable.
Ifacablefailureissuspected,theLinuxOSutilityethtoolshouldnotdetectalink:
[root@dm01adm01~]#ethtooleth0|grepLink
Linkdetected:yes

AnothersymptommightbethelinkspeedisnotwhattheNICisratedfor.Forinstance,iftheNICissupposedtobe
1Gb/sEthernet,itmightonlycomeupas100Mb/sEthernet:
[root@dm01adm01~]#ethtooleth0|grepSpeed
Speed:100Mb/s

NOTE:KeepinmindthattheiLOM's'inExadataareonlyratedfor100Mb/s.
Withthisinformation,onecanvalidateifitisduetoacablefailurebygoingtotheCiscoswitchandissuetheTDR
test.Thiswillrequiretogettothe"enable"promptintheswitch.
dm01swadm0#testcablediagnosticstdrinterfaceGigabitEthernet1/3
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrlstate=16sjohapiu_9

1/4

24/04/2015

DocumentDisplay

Thecommanddoesnotreturnanythingasyoujuststartedthetest.Itwillbeatthispointtheportbecomesnon
responsivetothehost.Tofindtheoutputorstatusofthetest,runtheshowcommand:
dm01swadm0#showcablediagnosticstdrinterfaceGigabitEthernet1/3
TDRtestisinprogressoninterfaceGi1/3
dm01swadm0#showinterfaceGigabitEthernet1/3|includelineprotocol
GigabitEthernet1/3isdown,lineprotocolisdown(notconnect)

Oncethetestiscompleted,toviewtheresultsusetheshowcommand:
dm01swadm0#showcablediagnosticstdrinterfaceGigabitEthernet1/3
InterfaceSpeedLocalpairCablelengthRemotechannelStatus
Gi1/31Gbps12N/AUnknownTerminated
36N/AUnknownTerminated
45N/AUnknownTerminated
78N/AUnknownTerminated

Inthiscase,thecablewasfineasitsStatusshowstheallthetwistedpairswereproperlyterminated.Terminated
meanstheimpedanceonthewireisasexpected.ItshowsthattheratedSpeedoftheinterfaceis1Gpbs,whichis
whatwouldbeexpected.Letstryonethathasafailure:
dm01swadm0#testcablediagnosticstdrinterfaceGigabitEthernet1/1
dm01swadm0#showcablediagnosticstdrinterfaceGigabitEthernet1/1
InterfaceSpeedLocalpairCablelengthRemotechannelStatus
Gi1/1100Mbps12N/AUnknownTerminated
36N/AUnknownTerminated
45N/AUnknownTerminated
78N/AUnknownTerminated

InanormalExadatainstallation,theDescriptionfieldintheswitchisfilledoutwiththehostnameofthenodeplugged
intotheport.Toseethisname:
dm01swadm0#showinterfaceGigabitEthernet1/1|includeDescription
Description:dm01cel14NET0

Tovalidatetheconnection,gathertheMACaddressfromtheExadatasystem:
[root@dm01celadm14~]#iplinkshoweth0|greplink
link/ether00:14:4f:ca:ef:cebrdff:ff:ff:ff:ff:ff

ToseewhattheswitchseesastheMACaddressoftheNICpluggedintotheport:
dm01swadm0#showmacaddresstableinterfaceGigabitEthernet1/1|includeGigabitEthernet
30580014.4fcb.1bc0dynamicipGigabitEthernet1/1

Wellthisisfun,thetwodon't'match.Thisisatestsysteminalab,soitmaynothavebeenleftinanormal
productionstate,whichisgoodforthisnote,badifthisisaproductionsystemandits3AMinthemorning.Tofindthe
properMACaddress,sshintoeveryhostandissue"iplinkshoweth0|greplink".
root@dm01adm08link/ether00:14:4f:cb:1b:c0brdff:ff:ff:ff:ff:ff

OncetheproperMACaddressisfound,onthathostcheckthestatusoftheNICusingethtool:
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrlstate=16sjohapiu_9

2/4

24/04/2015

DocumentDisplay

[root@dm01adm08~]#ethtooleth0
Settingsforeth0:
Supportedports:[TP]
Supportedlinkmodes:10baseT/Half10baseT/Full
100baseT/Half100baseT/Full
1000baseT/Full
Supportedpauseframeuse:Symmetric
Supportsautonegotiation:Yes
Advertisedlinkmodes:10baseT/Half10baseT/Full
100baseT/Half100baseT/Full
1000baseT/Full
Advertisedpauseframeuse:Symmetric
Advertisedautonegotiation:Yes
Speed:100Mb/s
Duplex:Full
Port:TwistedPair
PHYAD:1
Transceiver:internal
Autonegotiation:on
MDIX:off
SupportsWakeon:pumbg
Wakeon:g
Currentmessagelevel:0x00000007(7)
drvprobelink
Linkdetected:yes

Soitshowsthesameastheswitch,thatthespeedoftheNICis100Mbps.YetthisNICisabletoadvertise1Gbps.
Theswitchportisconfiguredfor:
dm01swadm0#showinterfaceGigabitEthernet1/1|includeHardware
HardwareisGigabitEthernetPort,addressis0024.9723.52c0(bia0024.9723.52c0)
dm01swadm0#showinterfaceGigabitEthernet1/1|includemedia
Fullduplex,100Mb/s,linktypeisauto,mediatypeis10/100/1000TX

NOTE:Don'tmindthehardwareaddresshereisdifferent,asitistheswitchport'shardwareaddressnottheNIC
intheExadatasystem.
ChangetheNICspeedusingethtool:
[root@dm01adm08~]#ethtoolseth0speed1000
Cannotadvertisespeed1000

Checktoseeifthechangetookusingethtool:
[root@dm01adm08~]#ethtooleth0|grepSpeed
Speed:1000Mb/s

Checktheswitchportconfiguration:
dm01swadm0#showinterfaceGigabitEthernet1/1|includemedia
Fullduplex,1000Mb/s,linktypeisauto,mediatypeis10/100/1000TX

Itlookslikethingsarefixed.Sohowdiditchange?TryTDRagainandseewhatitknowsabout.
dm01swadm0#testcablediagnosticstdrinterfaceGigabitEthernet1/1
dm01swadm0#showcablediagnosticstdrinterfaceGigabitEthernet1/1
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrlstate=16sjohapiu_9

3/4

24/04/2015

DocumentDisplay

InterfaceSpeedLocalpairCablelengthRemotechannelStatus
Gi1/1100Mbps12N/AUnknownTerminated
36N/AUnknownTerminated
45N/AUnknownTerminated
78N/AUnknownTerminated

Wait,itwentbackto100Mbpsandethtoolagrees:
[root@dm01adm08~]#ethtooleth0|grepSpeed
Speed:100Mb/s

ItappearsthereissomekindofproblemwheretheTDRtestiscausingthespeedtobeknockeddownto100Mbps.
Theredoesnotseemtobeacablefault,butseemstobesomekindoffaultineithertheCiscoswitchortheNICas
theinformation,atthispoint,isinconclusive.ThisnoteisnotaboutfindingabugineithertheCiscoswitchorNIC,but
howtouseTDR.
AnotherexampleshowsanactualfaultconditionusingTDR.Iftherebeenafaultinthecable,theStatusintheoutput
oftheTDRcommandwouldshow"Fault".OnthisswitchIexaminedoneportandfoundthis:
dm01swadm0#testcablediagnosticstdrinterfaceGigabitEthernet1/2
dm01swadm0#showcablediagnosticstdrinterfaceGigabitEthernet1/2
InterfaceSpeedLocalpairCablelengthRemotechannelStatus
Gi1/2100Mbps12N/AUnknownTerminated
36N/AUnknownTerminated
451+/10mUnknownFault
781+/10mUnknownFault

Eventhoughthisisa1Gbpsportintheswitch,a100MbpsNIConlyusesthefirsttwotwistedpairs.Herethespeedis
100Mpbsandthefirsttwotwistedpairsareproperlyterminated.Thecable"fault"isabouttheproperlengthof1
meter(eventhoughithasanuncertaintyof+/10m)whichshowstheTDRsignalbouncingbackfromtwopairsofa
1mcable.MorethenlikelytheNICinthiscasewasaniLOM,whichisonlyratedat100Mbps.

REFERENCES
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12
2SX/configuration/guide/book/intrface.html#wpmkr1066782
Didn'tfindwhatyouarelookingfor?

https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrlstate=16sjohapiu_9

4/4