12/30/2015

VMwareKB:Broadcom5719/5720NICsusingtg3driverbecomeunresponsiveandstoptrafficinvSphere

KnowledgeBase

Broadcom5719/5720NICsusingtg3driverbecomeunresponsiveandstoptrafficinvSphere(2035701)
Symptoms
Whenasystemusesthetg3driverwith1GbNICs,/var/log/vmkernelor/var/log/messageslogsreportmessagesindicatingthatNetQueuefeatureisenabledinthedriver:

T20:45:09.053Zcpu14:2091)<6>tg3:vmnic3:RXNetQallocatedon1
T20:45:09.053Zcpu14:2091)<6>tg3:vmnic3:NetQsetRXFilter:1[00:50:56:7f:96:940]
T20:45:44.054Zcpu7:2091)<6>tg3:vmnic3:NetQremoveRXfilter:1
T20:45:44.054Zcpu7:2091)<6>tg3:vmnic3:FreeNetQRXQueue:1
OneormoreNICsinthesystemstopfunctioningorresponding,reportingpartialorfulllossofnetworkconnectivitytovirtualmachinesoranyothertypeofVMkernelnetworking(vMotion,management,
NFS,iSCSI,etc).
TheNICsimpacteddonotappeartobereceivingCDP(CiscoDiscoveryProtocol)informationfromtheupstreamphysicalswitch.

Cause
ThisissueoccurswhenBroadcomBCM5719andBCM5720NICsareusedinthesystem.
Note:TheaforementionedNICmodelsarethemostcommonlyaffectedbutthisissuecanoccurwithanyNICthatusesthetg3driver.

Resolution
ThisissueisresolvedbyupdatingtheBroadcomdriver:
ForESXi5.5,downloadthelatestBroadcomtg3asyncdriver,availableatVMwareDownloads(https://my.vmware.com/web/vmware/details?downloadGroup=DTESXI55BROADCOMTG33133DV551&productId=353).
ForESXi5.0/5.1,thisissueisresolvedinBroadcomtg3asyncdriverversion3.129d.v50.1andlater,availableatVMwareDownloads(https://my.vmware.com/web/vmware/details?downloadGroup=DTESXI50BROADCOMTG3
3129DV501&productId=229).
ForESXi/ESX4.x,thisissueisresolvedinBroadcomtg3asyncdriverversion3.129d.v40.1andlater,availableatVMwareDownloads(https://my.vmware.com/web/vmware/details?downloadGroup=DTESXI4XBROADCOMTG3
3129DV401&productId=230).
Todetermineyourdriverversion,seeDeterminingNetwork/StoragefirmwareanddriverversioninESXi/ESX4.xand5.x(1027206)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1027206).
Formoreinformationonupdatingnetworkdrivers,see:
InstallingasyncdriversonESXi5.0/5.1/5.5(2005205)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2005205)
InstallingasyncdriversonESXi4.xandESX4.x(1032936)(/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1032936)
Note:Ensuretocontactandconfirmwithyourhardwarevendorbeforeupgradingtothedriverversionsmentionedinthisarticle.
Toworkaroundthisissuewithoutupgradingthenetworkdriver,disabletheNetQueuefeature.
Note:NetQueuecanonlybedisabledonthirdpartyasyncversionsofthetg3driver.InboxdriversarenowincludedwithESXi5.0Update2andESXi5.1anddonotincludetheNetQueuefeature.Toseethe
variousasyncandinboxdriverversionsfortheBroadcom5719/5720adapter,seeVMwareHardwareCompatibilityGuide(http://www.vmware.com/resources/compatibility/detail.php?
deviceCategory=io&productid=19946&deviceCategory=io&keyword=5719&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc).
TheperformanceenhancementfromNetQueuedoesnotbenefit1GbNICs.ThisfeaturespreadsthenetworkloadacrossmultipleCPUs,andasingleCPUcanhandleapproximately3Gbofnetworkload.
Therefore,ifthereareno10GbNICsonthehost,youcandisableNetQueueforthehostbyrunningthesecommands:
OnESXi5.x,runthiscommand:

#esxclisystemsettingskernelsetsnetNetqueueEnabledvFALSE
#reboot
OnESXi/ESX4.x,runthiscommandtoverifytheexistingsettingsonthetg3driver:

#esxcfgmoduleq|grepE"^tg"
or

#esxcfgadvcfgjnetNetqueueEnabled
netNetqueueEnabled=TRUE
Ifthereare10GbNICsonthehostinadditiontothetg3NICs,thenonlydisableNetQueueforthetg3driver.
TodisableNetQueueonESXi/ESXforthetg3driver,runthiscommand:

#esxcfgmodulesforce_netq=0,0,0,0tg3
TodisableNetQueueforthehost,runthiscommand:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035701

1/3

12/30/2015

VMwareKB:Broadcom5719/5720NICsusingtg3driverbecomeunresponsiveandstoptrafficinvSphere
esxcfgadvcfgkFALSEnetNetqueueEnabled
ToenableNetQueueforthehost,runthiscommand:

esxcfgadvcfgkTRUEnetNetqueueEnabled
Note:Thenumberofzeroes(0)intheforce_netqparameterarraymustbethesameasthenumberoftg3devicesonyoursystem.Forexample,theprecedingcommandappliesifyouhave4tg3NICs,which
canbeverifiedusingtheesxcfgnicslistcommand.
TorevertthechangeortoenableNetQueueforthetg3driver,runthiscommand:

#esxcfgmodules""tg3
Afterthechangesarecomplete,rebootthehost.

AdditionalInformation
Youcanverifythisissuebyunloadingandreloadingthetg3driverbyrunningthesecommands:
Tounloadthedriver:
#vmkload_modutg3
Toreloadthedriver:
#vmkload_modtg3
Tocheckifthesettingisconfigured:
1. Viewthecontentsoftheesx.conffilebyrunningthiscommand:
#cat/etc/vmware/esx.conf
2. Attheendofthisfile,ensurethatyouseeanentrysimilarto:
/vmkernel/module/tg3/options="force_netq=0,0,0,0"
ToverifythecurrentNetQueuestatusafteritisdisabled,runthiscommand:
#esxclisystemsettingskernellist|grepinetqueue
netNetqueueEnabledBoolEnable/DisableNetQueuesupport.FALSEFALSETRUE
Wherethekeysare:
BoolColumn=FALSE
EnableColumn=FALSE
disableColumn=TRUE
YoucanalsousethevSphereClienttomaketheconfigurationchange:
1. ClickthehostinvCenterServer.
2. ClickConfiguration.
3. UnderSoftware,clickAdvancedSettings.
4. ExpandVMkernelinthelistandclickBoot.
5. ScrolldowntothesettingnamedVMkernel.Boot.netNetqueueEnabledanddeselectittodisable.
6. Rebootthehost.
InESXi5.0andlaterversions,ifyoucanidentifythespecificNICthatismalfunctioningyoucanresolvetheissuebyforcingthelinkstatedownandthensettingitbackupattheOSlevel.Runthesecommands(to
resetvmnic1inthisexample):

localclinetworknicdownnvmnic1
localclinetworknicupnvmnic1
ThishasanadvantageoverunloadingandreloadingthedriverbecausethisonlyaffectsaspecificNICatatimeandnotallNICsusingthetg3driver.
Note:For10GBnetworkadaptersconnectingtoa1GBswitch,theswitchinterfaceshouldbelefttoautonegotiate.Contactyourswitchvendortoverifythatyourswitchsupports10GBNetworkInterfaces.

SeeAlso
DeterminingNetwork/StoragefirmwareanddriverversioninESXi/ESX4.x,ESXi5.xandESXi6.x(1027206)(http://kb.vmware.com/kb/1027206)
InstallingasyncdriversonESXi4.xandESX4.x(1032936)(http://kb.vmware.com/kb/1032936)
InstallingasyncdriversonVMwareESXi5.xandESXi6.0.x(2005205)(http://kb.vmware.com/kb/2005205)
tg3Broadcom5719/5720NICvSphere(2080266)(http://kb.vmware.com/kb/2080266)
tg3Broadcom5719/5720vSphere(2093995)(http://kb.vmware.com/kb/2093995)

UpdateHistory
10/18/2012Addedcommandstounloadandreloadthedriver
11/22/2012AddedthecommandforESX/ESXi
03/08/2013AddedlinkstodriverupdatesforESXi/ESX4.xand5.x.
02/02/2015AddedanotetoCausesectionregardingaffectedNICmodelsthatusetheTG3driver

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035701

2/3

12/30/2015

VMwareKB:Broadcom5719/5720NICsusingtg3driverbecomeunresponsiveandstoptrafficinvSphere

RequestaProductFeature
TorequestanewproductfeatureortoprovidefeedbackonaVMwareproduct,pleasevisittheRequestaProductFeature(http://www.vmware.com/contact/contactus.html?department=prod_request)page.
Permalinkto:Broadcom5719/5720NICsusingtg3driverbecomeunresponsiveandstoptrafficinvSphere(http://kb.vmware.com/kb/2035701)
Readourblog(http://blogs.vmware.com/kb/)

WatchKBTV(http://blogs.vmware.com/kbtv/)

Followus(http://www.twitter.com/vmwarekb)

RequestNewContent(http://www.vmware.com/landing_pages/knowledgebasecontent

request.html)

1(javascript:void(rate(1,this)))
2(javascript:void(rate(2,this)))
3(javascript:void(rate(3,this)))
4(javascript:void(rate(4,this)))
5(javascript:void(rate(5,this)))
85Ratings

Actions
BookmarkDocument(javascript:addBookmark(actionBoxUtil.docURL,actionBoxUtil.docTitle))
PrintDocument(javascript:window.print())
SubscribetoDocument(/selfservice/microsites/microsite.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2035701&format=rss)
(http://www.addthis.com/bookmark.php?v=250&username=xa4b5f42f36e60a29e)

KB:2035701
Updated:Oct5,2015
Categories:
StaffPicks
Troubleshooting
Languages:
English
Product(s):
VMwareESX
VMwareESXi
ProductVersion(s):
VMwareESX4.0.x
VMwareESX4.1.x
VMwareESXi4.1.xEmbedded
VMwareESXi4.1.xInstallable
VMwareESXi5.0.x
VMwareESXi5.1.x
VMwareESXi5.5.x
VMwareESXi6.0.x

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2035701

3/3