3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

UnifiedComputingBlog
Randompostsaboutunifiedcomputinganddatacenter

8GbFibreChannelor10GbEthernetw/FCoE?
Update2011/01/31Iveaddednewthoughtsandcommentsonthesubjecthere:
http://www.unifiedcomputingblog.com/?p=234
Whichisbetter?Whichisfaster?
Ivebeenstuckonthisoneforawhile.Imtraditionallyapurefibrechannelkindofguy,so
IvebeenprettyconvincedthattraditionalFCwasheretostayforawhile,andthatFCoEas
muchasIbelieveinthetechnologywouldprobablybelimitedtotheaccessand
aggregationlayersforthenearterm.Thatis,untilitwaspointedouttometheencoding
mechanismsusedbythesetwotechnologiesandtheeffectivedataratestheyallowed.Im
notsurewhyitneveroccurredtomebefore,butithitmeliketheproverbialtonofbricksthis
week.
Firstoff,aquickreviewofencodingmechanisms.Anytimeweretransmittingorstoring
data,weencodeitinsomeformoranother.Generally,thisistoincludesometypeofa
checksumtoensurethatwecandetecterrorsinreadingorreceivingthedata.Iremember
thegoodolddayswhenIdiscoveredthatRLLharddriveencodingwas50%moreefficient
thanMFMencoding,andwithjustanewcontrollerandalowlevelformat,myold10MB
(yes,thatstenwhoppingmegabytes,kids.Askyourparentswhatamegabytewas.)
suddenlybecame15MB!Well,wereabouttoembarkonasimilardiscovery.
1,2,4,and8GbFibreChannelalluse8b/10bencoding.Meaning,8bitsofdatagets
encodedinto10bitsoftransmittedinformationthetwobitsareusedfordataintegrity.
Well,ifthelinkis8Gb,howmuchdoweactuallygettousefordatagiventhat2outof
every10bitsarentuserdata?FClinkspeedsaresomewhatofananomaly,giventhat
theyreactuallyfasterthanthestatedlinkspeedwouldsuggest.Original1GbFCisactually
1.0625Gb/s,andeachgenerationhaskeptthisstandardandmultipliedit.8GbFCwouldbe
81.0625,oractualbandwidthof8.5Gb/s.8.5*.80=6.8.6.8Gbofusablebandwidthon
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

1/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

an8GbFClink.
10GE(and10GFC,forthatmatter)uses64b/66bencoding.Forevery64bitsofdata,only
2bitsareusedforintegritychecks.Whiletheoreticallythislowerstheoverallprotectionof
thedata,andincreasestheamountofdatadiscardedincaseoffailure,thatactualnumberof
dataunitsthatarediscardedduetofailingserialization/deserializationisminuscule.Fora
10Gblinkusing64b/66bencoding,thatleaves96.96%ofthebandwidthforuserdata,or
9.7Gb/s.
So8GbFC=6.8Gbusable,while10GbEthernet=9.7Gbusable.EvenifIwasabletouse
allofthebandwidthavailableonan8GbFCport(whichisveryunlikelyattheserveraccess
layer),with10GErunningFCoE,Idstillhaveroomfor3gigabitEthernetclasslanes.
Howsthatforconsolidation?
10GbFChasthesameusablebandwidth,andwithouttheoverhead(albeitasmall2%orso)
ofFCoE,butyoudontgettheconsolidationbenefitsofusingthesamephysicallinkforyour
storageandtraditionalEthernettraffic.
Imsold.

April8,2010 dave Miscellaneous dcb,fcoe

52thoughtson8GbFibreChannelor10GbEthernetw/FCoE?

Marek.Z
May4,2010at7:07am

Hi,greatinfo.Simpleandclear.Thanks!
However,couldyouexplainalittlebitmoreabouttheformulasyouusedtocalculatethe
bandwidth?
Cheers!

http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

2/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

DaveAlexander
May8,2010at9:38am

HiMarek
Specificallywhichpartsareyoucuriousabout?
Dave

Marek.Z
May10,2010at1:17am

HiDave,
ImcuriousabouttheFCcalculation(8.5*.80=6.8).Wherethe.80valuecamefrom?And
howaboutthe96.96%ofthebandwidthusingthe64b/66bencoding?Couldyoushowthe
exactcalcuation?
Thanks!
Marek.Z

DaveAlexander
May10,2010at5:53pm

8b/10bencodinguses10bitstoencode8bitsofinformation.Soforevery8bitsofreal
data,thereare10bitsofdatasentacrossthewire,(8/10=.80)ora20%encodingloss.In
64b/66bencoding,forevery64bitsofrealdata,thereare66bitsofdatasentacrossthe
wire,(64/66=.96),orroughly4%encodingloss.

Hoover
November11,2014at7:56am

8Gbitisalredyinclusiveencoding.soyourbandwidthis8Gbitusable(andints10Gbit
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

3/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

spectrumreally).
10GbitEthernetis99.5%avaliable
BERis1018forFC
and1015forEthernet
soitmeansonFCyougetoneerroreverymonthwhereonethernetoneeveryday.
Thatsthedifferent.

emiliano
January23,2015at11:52am

heythatswrong.FC100themasterconstantforFCtechis100MByte/sec,translating
in1.0625Gbpsincluding8b/10bencoding.Thestandarddefinedexactsuplicationof
ratessoMarekisright.1.06258=5.5Gbpsfor800MByte/sec
WhenspeakingaboutFC10thisisdifferentastheytriedtomatchedfor10GigLAN,
(whichis10.313Gbps,(64b/66bencoding).HerewehaveforFC101200Mbyte/secat
10.520Gbpsphy.
Ihopethishelps.indeeditsaconfusionsincetheybrokethenormwith10GFC.nowis
coming16GFC.
Regardingtheusageof8GFCplus3GEthatsveryrelativetoimplementationyou
see:the8b/10bgearboxinsideaSFPcanbeusableforboth.stillfromPHYtoupperL2
levels(FC4forFCandLLCforEth)theyareFULLdifferentstuff,evenmoreyouneed
somewaytodeagregatethe3GEto1GEportssoundswear(hereireallydontknow
whattheydo)

Marek.Z
May11,2010at1:37am

Okay,Igetit

Thanksfortheexplanation.

Cheers!
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

4/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Marek.Z

GregMcDonald
July16,2010at4:10pm

ExactlywhatIwaslookingfor.Themathematicaldifferencebetween8GbFC&10GbFCoE
Thankyou

DaveAlexander
July18,2010at8:29am

Greggladitwasuseful.Thanksforreadingandcommenting!

Mike
July18,2010at2:50pm

Aaah,thethingsonefindsontheInternetwithGooglethanksforausefulsnippetof
informationtoaddtothenoIdontwantabl*dyFCSANinournewDCprojecto)
CheersM.

ThorstenStaerk
July23,2010at3:03am

Totallyunsureaboutthat.Howlikelyisabitfault?Themorelikelyabitfaultisthebetterisa
8/10encodingversusa64/66encoding.Plus,fibrechanneloverethernetsoundslikethe
protocoloverheadswouldaddup.Notclearformefromyourarticle.

DaveAlexander
July23,2010at12:43pm

Bitfaultsareveryunlikelyhencethewillingnesstomoveto64/66bencodinginboth
10GEthernetand10GFC.Rememberthatthe8b/10bencodingusedin1GEisaveryold
methodandcomesfromaperiodintimewhenbitfaultsweremorelikely(especiallyover
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

5/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

copper),anddataratesweremuchlower.Athigherdatarates(like10G),theencoding
mechanismbecomesmuchmoreimportant.
WithFCoE,youreaddingabout4.3%overheadoveratypicalFCframe.AfullFCframeis
2148bytes,amaximumsizedFCoEframeis2240bytes.Soinotherwords,theoverhead
hityoutakeforFCoEissignificantlylessthantheencodingmechanismusedin1/2/4/8G
FC.

AlSim
August11,2010at1:57pm

Dave,
Couldyoupointmetothesourcesofthisinformation?ImeantheoneabouttheFCand
FCoEencoding.
Thanks
Al

DaveAlexander
August13,2010at2:50pm

DoaquickGooglesearchforfibrechannelencodingand10gEthernetencoding.AsI
recall,Wikipediahassomegoodsummaries.

MightyMan
January11,2011at6:30am

HiDave,
Thanksforthemathsinthecomparison.Itooam(orwas)moreforrealFC(guessithasto
dowiththeoldSCSIaffinity

).

Butsomethingstruckmereadingyourcomment#10.
Imaybemistaken,butisthemax.framesizeinethernetnotrestrictedto15XXbytes?
So,evenifyouencapsulateFCinethernet,youwouldneedtosend2framesintheworst
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

6/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

case.Justtakingthisintoaccount,andtherebyfiguring2148towards1500,youllendup
withroughlythesamespeed,giventhe8bitencodingofFCandthe64bitofethernet(the
overheadonencodingisbalancedwiththelargerpacketsizenevermindotherprotocol
effectslikeanincreaseinprobabilityofcollisionsif2framesneedtobesent,orethernet
havingtherandomwaitonresend,therebybreakingdowninspeedtheheavierthelineis
used).2148/1522*6.8=roughly10.
Imustadmitthough,thatIhaventreallylookedintothedepthsoftheFCmechanismsbut
Irecallthate.g.a16Mb/stokenringbeingaboutasfastasa100Mb/sethernet,whenthe
linesgotplugged.
AmItotallyoffwiththereasonsImentioned?
Greetings,
K.

DaveAlexander
January11,2011at7:08am

FCoEutilizesjumboEthernetframesintheneighborhoodof2300bytesthisallowsafull
FibreChannelframetobedeliveredwithinasingleEthernetframe.Ethernetdoesnot
nativelyhaveanycapacitytoperformframesegmentation.
Also,inaswitchedEthernetenvironment(wehaventusedsharedmediahubs/halfduplex
environmentsinalongtime),theresnolongerachanceofcollision,randomwaitson
resend,etc.SotheperformancedoesntsufferthewaythatoldsharedmediaEthernet
behaves.

Brcdbreams
January18,2011at10:38am

Dave,
Ithinkyouhavegottenthingsabitconfused.Yousaid,So8GbFC=6.8Gbusable,while
10GbEthernet=9.7Gbusablewhichignoresthefactthatthebaudrateisspecificallysetto
eliminatetheimpactofencodingonthedatarate.Tobeclear,1GbFibreChannelwith
8b/10bencodingatabaudrateof1.0625Gb/scandeliverexactly1Gbpsofdataflow.The
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

7/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

ANSIT11standardsetcthebaudratewithfullappreciationofencodingschemeusedto
ensure1Gbpsofdataflowfor1GbpsFibreChannel.
Asthelinkrateincreasedto2,4and8Gbps,thebaudratealsoincreasedproportionallyso
thatyougetexactly2Gbps,4Gbpsand8Gbpsofdataflowatthoselinkrates.
Ifyourpointwas8GbpsFCcantdeliver8Gbpsofdatarateand10GbpsEthernet
supportingFCoEcandeliver10Gbpsofdatarate,thenyouareincorrect.

DaveAlexander
January18,2011at11:32am

IfImincorrect,Imabsolutelyopentobeingeducated.WhatamImissinginthemath?Ifa
1GbFClinkisactually1.0625Gb/s,Imseeinga6.25%allowanceforencoding,while
8b/10bencodingincursa20%overhead.So1.0625*.80=.85Gb/s.SinceeachFC
generationincreasesproportionally,8GbFCwouldbe.85*8=6.8Gb/s.Iftheresaflaw
inmylogicormymath,byallmeanshelpmeunderstandit.
Infact,indoingsomeadditionalresearchtomakesureImnottotallysmokingcrack(its
alwaysapossibility)IdugupthisWikipediaarticlelistingbitratesforamassivenumber
ofinterfaces:Listofdevicebitrates.Thatpagealsoshowsthe6.8Gb/sfigurefor8GbFC,
alongwiththeotheriterationsofFCtechnology.Ididfinditabitoddthatthechart
neglectstoshow10GbFC,butitusesthesameencodingas10GbEthernet,sothebitrate
wouldbethesame.
Really,theonlydownsidetoFCoEistheadditionaloverheadoftheFCoE/Ethernetheaders,
whichtakesa2148bytefullFCframeandencapsulatesitintoa2188byteEthernetframe
alittlelessthan2%overhead.Soifwetake10Gb/s,knockdownforthe64/66bencoding
(10*.9696=9.69Gb/s),thentakeoffanother2%forEthernetheaders(9.69*.98),wereat
9.49Gb/susable.Ofcourse,allofthisispredicatedonfullFCframesifthemajorityof
trafficissubmaximumframesizes,theoverheadcanclimb.Yourmileageandbandwidth
mayvary.

DaveAlexander
January18,2011at7:50pm

http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

8/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Imwonderingifpartoftheconfusionmightcomefromthetraditionalstoragementality
(myselfincluded)ofthinkingaboutstoragebandwidthintermsofmegabytespersecond
(MB/s)asopposedtotraditionalEthernetmentalitywhichthinksaboutbandwidthintermsof
megabitsorgigabits(Mb/sorGb/s).
Giventhattheres8bitstothebyte,anetwork(regardlessoftype)capableofdelivering0.8
Gb/s(like1GbFC)isequaltoabout100MB/s.(Dontbelieveme?UseGooglecalculator
type0.8Gb/sinMB/sintoGoogleandseewhatresultyouget)
So8GbFC,withitsresulting6.8Gb/susablebandwidth,doesinfactgiveyoumorethan
800MB/s.
Therealissueiswhenwestarttryingtorecalculatetheseonthefly,doingthe100MB/sis
1000Mb/sis1Gb/s(whichiswrong)andgettingtheassociatedconversionerror.Itsreally
100MB/sis800Mb/sis0.8Gb/s.Sotheresyour20%overheadrightthere.
Justthinkingoutloudherefolks.

Brcdbreams
January20,2011at10:44am

HiDave,
Okay,Ifoundthecorrectmath.IttookabittorummagearoundintheatticIcallmymind.
Hereisthereference,FibreChannel,GigabitCommunicationsandI/OforComputer
Networks,AlanF.Benner,McGrawHillSeriesinComputerCommunications,Copyright
1996.,pg24.
yieldinganeffectivedatatransferrateof:
2048[PL]x1[byte]
1.0625Gbpsx=100.369MByte/sec
2168[PL+OH]10[encodebits]
where:
PLPayload
OHFrameoverhead

http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

9/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Asthisshows,encodingaswellasframeoverheadarefullyaccountedforwhencomputing
theeffectivepayloadtransmissionrateinMBytes/sec.
Ihopethisisinformative.
Asyoupointoutcorrectly,boththeFibreChannelandEthernetusethesameencoding,
64/66,at10Gbps.
TothesubtitleofyourpostWhichisbetter,Whichisfaster,theshortanswerisIt
dependsonalotoffactors,notjusttheencodingscheme.IMHO,itisveryhardtoargue
againstaprotocolthathashad15yearsofoptimizationforaspecificspecializedtask,as
FibreChannelhasbeenoptimizedforstorage,onthetechicalmerits.Itsanentirelydifferent
thingtodiscussotherfactorswhicharenontechnicalandhowthemmayaffectthe
customerdecisionofwhatisbest.
Onward

Brcdbreams
January20,2011at10:47am

Hmmwellthatattempttotypeouttheformulawasbolix.Hereitisinashorterform:
1.0625x2048[PL]x1[byte]=100.369MByte/sec

2168[PL+OH]10[encodebits]

Brcdbreams
January20,2011at10:50am

DoubleHMMMMitseemswhitespacegetsstrippedinWordPressOneMoreTime.
1.0625x(2048[PL]/2168[PL+OH])x(1[byte]/10[encodebits])=100.369MByte/sec

vikram
October23,2013at1:20am
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

10/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

whatisplayloadifurrusing.mxffiles?

DaveAlexander
January20,2011at11:24am

Ifthegoalis100MB/s,Iwouldagreethatitsfullyaccountedfor.Thedifferenceisthatthe
conversationisntabout100MB/s,itsabouteffectivedatarate.Atheoretical1.0625Gb/s
linkusing64/66bencodingwouldprovide131MB/s.Iagreethat8GbFCprovides800MB/s
(actuallyabitmore).Butagain,theencodingmechanismhinderstheeffectivetransferrate.
ThatsthebeautyofFCoEtheprotocolspecificoptimizationsthatcomefromthelongterm
developmentofFibreChannelasaprotocolareretainedwithFCoE.Onlythewireis
replaced,alongwithanewflowcontrolmechanismthatreplacesbuffertobuffercreditswith
somethingthatapproximatesthesamebehavior.Theprotocolisntmodifiedinanyway.
IdontdisagreethatFibreChannelasaprotocolisoptimizedforstoragetrafficwhichis
whyIpreferitoverotherstorageapproaches(iSCSI,NFS,etc)regardlessofthewireits
runningover(FCorEthernet).
IsntBrocadeinvestingalotofmoneyinFCoEthemselves?

Brcdbreams
January20,2011at1:09pm

Dave,
Imnowthroughlyconfusedbyyourcomments.Sorry.Iguessitstheadvancedyearsand
greyhair
Sowhatdoyoumeanbyeffectivedatarate?IknowwhatImeanbyit.Pleaseseetheresult
intheformulaIprovidedwhichcomputeseffectivedatarate,for1GigFibreChannel
whichisthemaximumdatarateyoucanflowonthewire.ThatswhatImeant.
Youmustappreciatethatencodingschemesarecriticalchoicesatthephysicallayerofa
standard.Theyinfactdonothindertheeffectivetransferrate.,butareexplicitly
designedtoachievethatrateandtodosocosteffectively.How?
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

11/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Anencodingschemerequireshardwaretosupportitanithastobecosteffective.Pleasenote
thedateofmyreference,1996,whichisatthebeginningofthecommercialapplicationof
FibreChannel.Atthattime(15yearsago)1GbpswasFAST.The8b/10bencodingmethod
developedbyIBMwasefficient.Infact,itwassoefficientthatitcontinuedtoworkat8xthe
bandwidthandover15years.Thatsoutstanding.
At10Gbps,theEthernetandFibreChannelstandardsgroupsjointlyselected64b/66b
encodingsince8b/10bcouldnolongermeetthebiterrorraterequiredatthatspeed.Note,
BOTHapprovedit.Why?Well,toreducecostatthephysicallayer.
Ifyoudigdeeper,youwillnotethattheEthernetandFibreChannelbaudratesarenotthe
sameat10Gbps.Why?Ithastodowiththeframeformat.
Now,ifyouwanttodebateefficiencyofEthernetframesvsFibreChannelframesfor
transportingstroragedata,andsupportthecontentionthatEthernetismoreefficient,good
luck.Youwillfirsthavetoexplainhowaprotocol(Ethernet)requiringan8byteheader(pre
amble+SOFHeader)ismoreefficientthanonethatrequiresonly4bytes(FiberChannelSOF
primative).
Next,youwillhavetoshowwhyaddingtheFCOEshimlayerbetweentheFibreChannel
payloadismoreefficientthannotaddingitwhichisthethecasewithFiberChannel.Youare
incorrectwhenyoustateThatsthebeautyofFCoEOnlythewireisreplaced
ThereisanaddedFCoEshimintheframe,andthatsnotasefficientisit?
Now,toyourfinalquestion,yesBrocadeisinvestinginFCoE.Why?Becausethereareuse
casesandmarketsegmentsthatmakeeconomicsenseforthatinvestment.Butnotalluse
casesandmarketsegmentsmakesenseforFCoE,whichiswhywestillinvestinpureFibre
Channelincludingourgoaltodeliverthefirst16GbpsFibreChannelswitchintheindustry.
http://newsroom.brocade.com/article_display.cfm?article_id=1437
And,wearecontinuetoinvestheavilyinEthernettechnologydeliveringtheindustrysfirst
EthernetfabricwithournewBrocade6720VDXfamilyofdatacenterswitcheswhichwon
themostimportantenterpriseITproductoftheyearfromCIOEdge.
http://newsroom.brocade.com/article_display.cfm?article_id=1542
Onward.
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

12/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Brcdbreams
January20,2011at1:43pm

Dave,
IhappeneduponthisCSCOTechnicalBriefpublishedbackinthedaywhenEtherentat1
Gbpswasnew.
http://www.cisco.com/en/US/tech/tk389/tk214/tech_brief09186a0080091a8a.html#wp16040
NotetheleverageoftheANSIT11workon1Gbpsspeedsand8b/10bencodingforFibre
Channelatthephycicallayerofthestandard.NotealsothedifferentbaudratesforEthernet
vs.FibreChannel,duetoframedifferences,toachivethedesiredgoalfordatarate.
Hopeyoufindthatreferenceofsomeinterest.

DaveAlexander
January20,2011at4:36pm

NotreallysurehowmuchmoreplainIcanmakeit.Youseemintentonfocusingondesired
datarateasopposedtoactualperformance.Imnotquestioningyourfiguresyouseemto
beignoringmine.Yes,1.0625GbFCgives100MB/s.Yes,8.5Gb(1.0625*8)FCgives
800MB/s(orbetter).Nooneisdisputingthat.ItdoesntmatterifwecompareinMB/sor
Gb/s,8b/10bencodingwillalwaysbelessefficientthan64b/66bencoding.
SinceyouresofocusedonMB/s,letsdothenumbersthatway.1.0625GbFC=100MB/s.
Sothatsourbaseline.8.5GbFC=870MB/s(bypuremath).10GbFCoE=1214MB/s(by
puremath).Now,letsletsnormalizeforlinkspeed870/8=108MB/sforeachGb/sof
linkspeedprettyclosetowhatwestartedwith.1214/10=121MB/sforeachGb/soflink
speed.Thedisparitygetsevenlargerifyouusethe800MB/svaluefor8GbFC.
Thesenumberstakeintoaccountthevariousprotocoloverheads,etc.Idontreallymind
throwingafewextrabytesaroundinoverheadandencapsulationwhenImstillgetting13
MB/smorethroughputforevery1Gboflinkspeedandbandwidththatsflexiblebetween
protocols.ThatsafullextraGigabitEthernetlinkthereforfree,andthatsevenassuming
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

13/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

thatyouremaxingoutthe8GbFCbandwidthaHUGEassumptionattheaccesslayer,
whichiswhereFCoEhasthemostdramaticandimmediatepayback.Insteadofcablingby
protocol,whyonearthwouldntyoucableforbandwidthwithefficientencoding?
YourtonesuggeststhatImantiFibreChannel,whichcouldntbefurtherfromthetruth.Ive
beenalongtimesupporterofFCandcontinuetobeso.Obviouslysomethinghadtogivein
encoding,whichiswhyFCisfinallyadopting64b/66bwith16GbFC.Thebiggestreason
NOTtochangeencodingisthatitmakestheportsincompatiblewithpreviousversions.
Once16Gbhitsthestreet,theFCargumentchangesquiteabit.Ofcourse,with40Gb
Ethernetrightaroundthecorner,itdoesntchangeforlong.

Brcdbreams
January21,2011at12:24pm

Dave,
Heyman,nooffensenordisrespectwasmeantinmycomments,norwasitmyintentto
accuseyouofanyparticularbiasvisavismytone.FolkswhoknowmewilltellyouIm
bluntanddecidedtonedeaf
HopeyouhaveaGr8daymyfriend.

Max
May28,2011at6:44am

Thanks!
iwaslookingforthisexplanation.
simpleandclear!
ihaveaquestion:whataboutiSCSIon10GbE?
Max

Robert
May31,2011at3:25pm
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

14/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Ithinkyoucanhelpme.WhenusingthecommandportperfshowonaBrocadeFCswitch,
isitmyassumptionfromreadingyourblogthatona4GB/sHBAportifIsee425MB/sthen
thatmeansIamsaturatingthatport.
Whenusingwebtoolsonthebrocadeswitch,itneverdisplaysmorethem100MB/s
throughputsomyassumptionistheresabottlenecksomewhere.SinceImanoviceatthis,I
mustask.Bythewaytheswitchisnotdisplayinganyerrors.
Anyhelpwouldbeappreciated.

DaveAlexander
June1,2011at11:38am

@Maxintermsofthisdiscussion,iSCSIover10GEisjustnormalTCPtrafficlikeany
other.
@Robertyouvedefinitelygotabottleneck.100MB/sshouldbeabigclue,sincethats
rightat1Gb/sspeed.Soundslikesomethinginthepath,oroneoftheenddevices,is
operatingat1Gb.

BrianMcMullan
June30,2011at7:17am

ThirdI/OhaspublishedAnAnalysisof8GigabitFibreChannel&10GigabitiSCSItheir
analysissays8GbFibreisstillsuperiortoiSCSI.
http://www.emulex.com/artifacts/e261005e461e42b1ba7e0778297b2c34/thirdIO.pdf

Latif
September6,2011at12:16pm

IlookedatthatThirdI/Opdf,andIhavetosayitsamazingwhatresultsonecancookup
whenincontroloftheingredients.
Giventhebenefitofthedoubtthattheresultsareactuallywhatwasposted.Themajorflaw
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

15/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

andquestionIhavetoaskiswhyintheworlddidtheyuseWindows2003serverinthemix
atall?Itsan9yearoldOSwithasinglethreadedTCP/IPstackthatwasneverdesignedto
scale,andyeswhilepatchesandimprovementshavebeenmadetothatstack,itsinherently
flawedwithitsdesignfocusofprovidingnetworkservicesforslowerandcongested
networksliketheinternet.
IwouldbuywhattheyresellingiftheypostsomeLinuxornativeWindows2008to
Windows2008numberonthere.
Cheers

Latif
September6,2011at12:49pm

Also,asanaddendumtomylastpost,whydidtheyuseAMDCPUswhentheirpercore
performanceispoorwhencomparedtoIntelsWestmereline?
CoupleanAMDCPUandWindows2003singlethreadedTCP/IPstackandyouwillget
inferiorperformanceagainstaFCadaptereverytime.Itstheperfectrecipetobuildan
inferiorperformingsystem.
Lastly,thisconversationwillbemootrealsoon.BasedonroadmapconversationsIvehad
withOEMs,AllthemajorplayersaremovingtoaconvergedadapterthatwilldobothFC
andEthernetallonthesamecard,allyouhavetodoisenablethefeatureinsoftwareand
carveoutthebandwidth.Allthesepurposebuiltcardsandswitcheswillbephasedoutinthe
next5years.

HorstTruestedt
November6,2011at3:17pm

Helloall,
DaveisrightabouttheoverheadforFC,butthereissomeadditionalinformationnecessary.
FCfocusesonMB/sofcustomerdata.WedesignedFCtocarryat2kpayloads,100MB/sof
customerdatasimultaneouslyinbothdirectionsevenwhenusingoneACKperframe(the
additionaldatarateisusedtocarrytheoverheadof8b/10bandtheframeoverhead).When
westartedFCin1988,64b/66boverheadwasexpensivetoprocessvsthe8b/10b.Thebit
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

16/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

overheadisprimarilyforclocksynchronizationandsecondarilyforerrordetection.FCwas
alsodesignedtouseeverythingasa40bit(inthe10bcode)Wordtofacilitatehardware
synchronization,detection,etc.Anumberof40bitcodesareusedforflowcontrolandother
purposes(thesearetypicallydonewithpacketsinEthernet).So,thereissomeadditional
overhead.Asanexample,tomake10GEusefulforFC,werequiredsimilarreliabilityasin
FC.Oneoftheseisnodroppedpackets.Todothis,10GEusesthePAUSEflowcontrol.
So,onereallyneedstolookbeyondthebitratetoseewhattheactualperformanceisas
measuredbythecustomersdatathroughput.

DaveAlexander
November6,2011at3:20pm

Thanksfortheadditionalinsight,Horst!

HenryLongabaugh
January25,2012at4:03pm

HiDaveYouhaveexcellentknowledgeofFCthroughput,doyouhaveanyinformationon
themorephysicallayerforfibre?Likewhatwouldbetheluminescencetolerancesforthe
differencesbetweena0&1bit.And,howdoesthefibrechannelkeeptiming
synchronizationbetweenthesenderandthereceiver?

DaveAlexander
February7,2012at1:51pm

HiHenry
ThisiswellbeyondwheremyknowledgeofFCstops.IllseeifIcandigupsome
referencesforyou.
Dave

Kevin
March2,2012at10:41am
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

17/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

IBMranatestof4Gbfiberchannelvs10GbFCoEandtheyperformedaboutthesame.That
means8Gbfiberchannelwillperformabouttwiceasfastas10GbFCoE:
http://www03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101686

DaveAlexander
March21,2012at11:20am

Actually,whattheyprovedwasthatforthatparticularworkloadandconfiguration,theygot
aboutthesameperformance.Ifyouswappedout8GbFCfor4GbFCinthatsametest,I
wouldexpectittoperformaboutthesameasthe4GbFC.Ifyoulookatthetest,itshows
thatthetransportreallywasntthatimportantforthatworkload.Forwhattheywantedto
prove(whichisusewhateveryourecomfortablewith),theywerequitesuccessful.The
issueisntwhichoneisfasteritswhichonegivesmethemostusabilityANDflexibility.
ImalongtimefanoftraditionalFC,butIvebecomeamuchbiggerfanofflexibility.

Jason
March21,2012at9:55am

Thisisaninterestingtopicofdiscussion.IknowitsoldatthispointbutIdliketoknowif
youvereturnedtothistopicnowthat16GbpsFCisoutandusesthe64/66encoding?

DaveAlexander
March21,2012at11:27am

16GbsFCactuallyonlyusesalinkspeedofaround14Gbps.Theirtargetis1600MB/s,
not16Gbps.Ifindcallingit16GbFCverymisleading

Kent
May30,2012at11:01pm

InrespondtoKevinsclaimthatasingle4GbFCiscomparabletoasingle10GbFCoE,he
mightwanttoreadthewholearticlebeforecomingtoaconclusion,ormisleadingothers.

http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

18/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Thetestisbasedon8X4GbFC=32Gbpsbandwidthvs3X10GbFCoE=30Gbps
bandwidthAndFCoEisstillfasterdespitethelowermaximumbandwidth.

Bob
November2,2014at9:25pm

IhavetoagreewithKenthere.ThatswhatIreadalso.IntherealworldIworkinbothof
theseenvironments,andwhenitcomestomovinglargevolumesofdataaround,theFCoE
haswonhandsdown.ThatincludesIBM,DELLandHP,Ivebeenonallofthem.IBMI
mustsayhasalwaysbeentheslowestsystemsnotmatterwhatyouputonthemFCor
FCoE.SoIthinktheequationshouldalsoconsiderthesourcetobemoreaccurate.

toby
July20,2012at5:53am

ImsureIreadsomewherethattheTCPoverheadmeantthat10GiSCSIcouldntmatch8G
fibre.
ImnotanexpertandIvelookedandcantfindthereference.
Thisfactdidinfluencemydecisiontogowithfibrewhenchoosingastoragesolution.
(coupledwiththelackof10Ginfrastructure).

DaveAlexander
July20,2012at4:47pm

Ifmytwochoiceswere8GFCor10GiSCSI,Idprobablygowiththe8GFCtoobut
thatsmoreofaprotocolthanaspeedthing.EvenwiththeoverheadofTCP/IP,theyre
probablysimilarinperformanceatleastgoodenoughformostapplications.Idrather
havetheFCprotocoltoworkwithsinceImmorecomfortablewithitfroma
troubleshootingandmanagementperspective.

tom
July26,2012at12:27pm
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

19/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

Dave,
Ihavebeenreadingthisthreadandhaveadumbquestion?Ifthetransferratesofthedisksin
anarrayarelimitedto3Gb/sectransferrateoncetheygothroughacopperlink(ref.EMC
clariion480documentation).Doesntthatseemtomakeabigdependencyhowthedisk
groupsarelaidout?Couldyoupleasehelpmeunderstandthis?ifthereisare3disks
concatenatedinamirroredDoesntthatmeanthatthemaxdatatransferrateofdatawillbe
limitedto3Gb/secforaspecificread/writeoperationtoaservercorrect?Thanksforreading.

DaveAlexander
July29,2012at12:47pm

True,transportthroughputisonlyone(andsometimessmall)partoftheequation.Ifyoure
talkingaboutamirroredset,yourerightyoudbelimitedonanyonetransactiontothe
speedofthediskandthatsassumingthatparticularspindleisntservinganyother
requestsatthesametime.

Pingback:Technical:StoragePlanningDeterminingStorageRequirementsbasedonIOPS
Requirements|DanielAdeniji'sLearningintheOpen

Pingback:Confluence:IxNetwork

Pingback:Confluence:IxNetwork

Bob
August26,2014at10:11am

Nicepost,butIdliketopointoutthattheextrabitsin8b/10band64b/66bencodingarenot
therefordataintegrity.Theyarethereforclockrecoveryandtobalancethebitstream.They
areusedtokeepthenumberof0and1bitsapproximatelyequalwhichisimportantfordata
transmissionequipment,andtolimitthenumberofsuccessive0bitsinarowsothatclock
recoveryiseasierinthehardware.Thesebitscanbeusedtodetectlineerrors,ascertaincode
sequencesareillegal,buttheyarenotusedinanytypeoferrorcorrection.
http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

20/21

3/6/2015

8GbFibreChannelor10GbEthernetw/FCoE?|UnifiedComputingBlog

ProudlypoweredbyWordPress

http://www.unifiedcomputingblog.com/2010/04/08/8gbfibrechannelor10gbethernetwfcoe/

21/21