MakingSenseofBroadbandPerformance

UnderstandingInternetSpeedTestResults
WhyDoSomeSpeedTestResultsNotMatchMyExperience?
YouonlyhavetoreadthepopularpresstoknowthattheInternetisaverycrowdedplacetoworkor play.ManyInternetServiceProviders(ISPs)aretalkingaboutchangingfromamonthlysubscription modeltoausagemodel,muchinthewaythatanaloguepublictelephoneservicehaschangedover time.Itisnotsurprising,therefore,thatmostInternetusersencounterperformancerelatedservice problemsonafairlyregularbasisandwanttovalidatethattheirISPsaredeliveringthecontracted serviceperformance. Asaresultthereisamultitudeofspeedtestingwebsitestobe foundontheInternet.Someofthesetestingservicesareprovided bytheISP,whilemanyareprovidedbyindependentthirdparties. Naturallythekeyquestionis,Isthespeedtestaccurate?Andif thespeedtestresultsdonotmeetexpectations,doesthetester providethedatatoresolvetheunexpectedresults?

Theproblemisnotinthe measurement,itisin understandingthetest resultsastheyrelateto theapplicationproblem beingexperienced.

Unfortunately,allspeedtestsarenotcreatedequalandthe testingapplicationsrarelydefinethemethodusedtoconductthe test.Withoutunderstandingthetestingmethodology,validating theresultsisadifficultorimpossibletaskfortheuser,regardless oftheirskilllevel.ThecomplaintmostechoedaroundtheInternetdiscussiongroupsis,"Whyisthis videodownloadtakingsolongwhenmyspeedtestershowsIgetmyfull10Mbps?"

Theproblemisnotinthemeasurement,itisinunderstandingthetestresultsastheyrelatetothe applicationproblembeingexperienced. AbetterunderstandingofInternetspeedtestmethodscanbegainedthroughcomparisonstovehicular traffic.Forexample,ifthelocalairportis60milesawayandtheroadspeedtotheairportis60milesper hour,ifyouput4passengersinyourcaranddrovetotheairportyouwouldmostprobablyexpressthat speedas4passengersperhour.Ifyounowrentedabusandfilleditwith50passengersanddrovethe exactsamejourney,youcouldreportyourspeedas50passengersperhour.However,thelocal authoritythatownstheroadtotheairportmightpublishtheroadscapabilityas50,000passengersper hour.Thisdifferenceliesinlocalauthoritysassessmentofpassengerthroughputfortheroadat maximumcapacitywithallthepassengersinallthecarsandbusesaddedtogether,thenusingthat totaltoexpressthepassengersperhouroftheroad.Eachofthese3differentmeasureshasvalidity withinthecontextofthetestconducted.However,eachtestmethodiscompletelydifferentinthe

MakingSenseofBroadbandPerformance

valueitdeliversfortheindividualapplicationuser.OneofthebiggestfailuresofInternetspeedtesting applicationsistheirinabilitytodifferentiateatrueapplicationspeedtestfromalessmeaningful capacityspeedtest.Whatisthedifference? Asanexample,alocaltelecomproviderannouncedinthemajorpressrecentlythattheywouldsoon offer100Mbps(100megabitspersecond)connectionstothehomeanddownloadspeedswillbe2030 timeswhattheyaretoday.Exampleswerequotedofamoviedownloadtakingonlyminuteswhereas existingservicesaremeasuredinhours.Theproblemwiththisstatementisthatitcanbecompletely misleadingjustbecauseanInternetconnectionisratedas100Mbpsdoesnotmeanyouwillgeta 100Mbpsspeedonyourconnection. Justlikethesimpleroadexampleabove,wherethelocalroadauthoritypublishedthespeedat50,000 passengersperhour,thespeedofaconnectionisbeingpublishedat100megabitspersecond.The problemwithspeedtestersthatmeasurecapacityspeed,assomeofthemostpopulartestersdo,isthat thecapacityofaconnectiondoesnotrelatetotheapplicationspeedoftheconnection.Theresultof thisdisconnectisthattheuseroftheconnectiongetsveryfrustratedwiththeactualspeedachieved becauserealitydoesnotmatchthepublishedexpectation. Intheroadspeedtestexample,thebusapplicationwasonly50passengersperhour,thecarswasonly 4passengersperhour,thereasonforthisisobvious.However,tounderstandtheimportanceofan applicationspeedtestitisfirstnecessarytounderstandsomeoftheprinciplesofwhytheInternetis designedthewaythatitis. First,theInternetisoftendescribedasacontendednetworkaswellasabesteffortnetwork.A contendednetworkmeansthatalltheuserscontendforuseoftheInternethighway,notunlikecars contendingforuseofthetraffichighway.Besteffortmeansthatinacontendednetworkthereareno guaranteesthatyourapplicationdatawillgettothedestinationinatimelymanner,orevenatall. Withtheselimitationsinmind,theInternetwasdesignedtocopewiththestressesandstrainsof contention.However,todothismeantthattheprotocolsthatdrivetheinternethadtoincorporateflow control.WithoutsomeelementofflowcontroltheInternetwouldsimplynotwork,itwouldcollapse underthestressesofthedataasfastconnectionsjoinedslowerconnections.Inourroadspeed example,ifyouaredrivingtotheairportwithyour4passengersandarriveatajunctionthathasvery heavytraffic,yourabilitytoenterthattrafficflowwillbedependentontherebeingagapbetweentwo vehicles,orpossiblyasetoftrafficlightsthatgiveyoupriorityatacertainpointintime(regulation).Of courseiftrafficissoheavyatthejunctionthattherearenogaps,andtherearenoregulatorylights, thenitisunlikelythatyouwillachieve4passengersperhourandyoumaynotgettotheairportatall.

2 2008VisualwareInc.

MakingSenseofBroadbandPerformance

IntheInternetworldbytesarenotmeasuredinvaluesaslowas4oreven50,asinthecar/busexample. InsteadtheInternetdealswithnumbersofdatabytesthataremanyordersofmagnitudegreater.For example,adownloadofamusicfilecanbemeasuredinhundredsofmillionsofbytes.Tohelpresolve thetwomainissuesofcontentionandbestefforttheInternetmanagementprocesssendsdatain limitedchunksatanyonetime,afterwhichthesendingcomputerwaitstohearfromthereceiving computerthatthedatahasarrivedbeforesendingmoredata.Thisprocedureallowstheprocessto ensuretheintegrityofthedata,aswellasrecoveryifdataislost.Takingthisapproachdeliverstwovery importantcharacteristicstoperformance:1)Theperformanceoftheconnectionmustincludethe returnjourneytimebacktothestartingpointbecauseconfirmationisrequiredbeforemoredatacanbe sent,and2)Thesegmentationofthedatachunksinthismannereasestrafficcongestionbyeffectively creatingnaturalgapsinthetraffic. Giventhisconsiderationandapplyingittotheairportcar Theperformancethroughput example,ifyouhadmorethan4passengerstodriveto ofanyconnectionwilllargely theairportinyourcar,thepassengersperhourspeed wouldnotbe4aspreviouslystated,but2perhourasthe dependonthedistance totaljourneytimewouldneedtoincludethehourtothe betweenthestartingpoint airport,andanadditional1hourreturnjourneytocollect andthedestination,coupled thenextlotofpassengers.Akeypointtounderstandin withthesizeofthevehicle thisanalogyisthattheperformancethroughputofany connectionwilllargelydependonthedistancebetween usedtocarrythepassengers. thestartingpointandthedestination,coupledwiththe sizeofthevehicleusedtocarrythepassengers.Thebus, forexample,wouldprovidebetterpassengersperhourperformancebecauseitcouldtake50 passengersatatime. HoweverweneedtodelvefurtherintoInternetperformancecharacteristicstobetterunderstand measurementsofthroughputandspeed. Atthispointyoumightthinkgreat,thenwewillhaveabusthatcantakemillionsofpassengersata time,notacar.Thisisapossibility,intheory,howeverthisapproachwillbeimpactedbycontention becausethechunkofdatawouldbesomuchlarger.Inadditionthereistheissueofwhathappenswhen thevehicledoesnotmakeittotheairportbecauseoftheresultingcontentionissues.IntheInternet worldthisiscalledpacketloss.Eachchunkofdatasentisbroken intosmallerchunkscalledpacketsandpacketsmaynotreachthe Whenpacketlossoccurs, destinationbecauseofcontentionissues.

chaosreigns.

Whenpacketlossoccurs,chaosreigns.Thereceivingendhasto notifythesendingendthatapackethasnotarrived.Therecanbe manydifferentreasonsforthis,butregardlessofthereasonthesendingendhastobetoldtoresend themissingpacket(s)ofdata.Theamountofchaoscausedwilldependonwherethelostpacketwasin thechunkofdataandjusthowmanypacketswerelost.

3 2008VisualwareInc.

MakingSenseofBroadbandPerformance

Oneveryimportantreasonforthechaoswithregardtotheapplicationisthatdatahastobeprocessed inorder.Ifapacketislostatthebeginning,thenthereceivingendcannotprocessthesubsequent packetsuntilthemissingpacketorpacketsarerecovered.Sothosepacketsthatfollowthemissing packetmustbestoreduntilthemissingpacketsareresentandtheapplicationthatwantsthedatahas towait.Thiscanhappenseveraltimeswiththesamechunkofdata,sothelargerthechunkthelarger therisk. Youmightnowthink,ratherthansendoneverybigbusofpassengerslet'ssendtwosmallerbusesor evenmore,let'ssend4busses.Unfortunately,thisdoesnotaddresstheproblembecausemostcritical Internetapplicationsrequirethedatatobereceivedinorderandbusnumber2couldarrivebeforebus number1.Realtimevideoisagoodexample,imaginewatchingavideowhenframenumber100 appearsbeforeframenumber10.Forvideoapplicationsandevenfinancialapplications,suchasstock trading,datamustbeprocessedintheordersentfortheapplicationtofunctioncorrectly.Some applicationscansupportmultiplebusestomovedatabutapplicationsthatcanacceptdatainanyorder andstillfunctioncorrectlyarenotthatcommon.Webpagesareagoodexampleofanapplicationthat canacceptdataoutoforder,forthisreasonitisnotuncommonforitemsatthebottomofthepageto appearbeforeitemsatthetop.Howeverthisonlyworksbecauseitdoesnotaffectthewebpageusage. Avitalquestionthenis,"Whydoesthismattertoaspeedtest?" Itmattersahugeamount,andthereinliestheproblemofunderstandingtheresultsofaspeedtest. TheInternetdeliversawiderangeofapplicationstotheuser,beit listeningtomusic,watchingavideo,browsingawebsiteortrading stocks.Eachapplicationwillmakeuseoftheconnection differentlyandtheperformanceachievedwillbesubjecttothe applicationrequirementsandusagemodel.Aspeedtestthatdoes notinvokeatestmethodthatmatchestheapplicationusagewill notdeliverameasurethatwillreflecttheactualperformanceof theconnectionasitrelatestothatapplication.Thisoversightby speedtestingapplicationsissingularlythecauseofmoreuser frustrationthananywhentryingtounderstand: "Whyisthisvideotakingsolongtodownloadwhenmyspeed testershowsIgetmyfull10Mbps?". Letsnowexaminethedifferenttypesofspeedtestthatexistand whytheymostlyfail.

Aspeedtestthatdoes notinvokeatestmethod thatmatchesthe applicationusagewillnot deliverameasurethat willreflecttheactual performanceofthe connectionasitrelatesto thatapplication.

TheCapacityTest
OneofthemostpopulartestapplicationsontheInternetinvokesacapacitytest.Inotherwordsitis guiltyofsendingmultiplebusestotheairportallfullofpassengers.Asaresultthetestingapplication effectivelyfillseverylaneontheusersInternethighway.Thetestingapplicationcantakethisapproach 4 2008VisualwareInc.

MakingSenseofBroadbandPerformance

primarilybecauseitdoesnotcareaboutthedistancetravelled,thepacketorderoreventhepackets thatgetlostbecauseoftheincreasedcontention.Thetestsimplyreportsthecapacityspeedofthe connectionbasedonthevolumeofdatathatsurvivesthejourneyandarrivesatthedestination.Asthe speedtestingapplicationdoesnotcareabouttheorderofthedataorthetriptimeforaconnection,it deliversanincorrectandmisleadingresult.Thisistheequivalentoftheairportroadauthoritypublishing aspeedof50,000passengersperhour. Unfortunately,unlessthetestinguserrealizesthatthetestapplicationissendingmultiplebusesofdata simultaneouslytoprovideacapacityspeedmeasure,theywillgetveryfrustratedwhentheirvideo application,whichhastomaintaindataorderandrecoveralllostpacketstoprovideaqualitymovie experience,failstomatchtheperformancelevelreported.Hencethequestion: "WhyisthisvideotakingsolongtodownloadwhenmyspeedtestershowsIgetmyfull10Mbps?".

TheCompressionTest
AnotherpopularmethodusedinspeedtestingapplicationsistoprovideanHTTPdownloadofalarge file,usuallyajpegimageorequivalent.Thisisasimplebutcommonapproach.Thetestinglogicreads andtimesthedataandreportsthespeedaccordingly.Onthesurfaceyouwouldexpectthatexpressing dataovertimewoulddelivergoodresults,however,becausethejpegfilecanbecompressedthe amountofdatasentmaybesmallerthanthedatabeingread.Compressionratioscanbeashighas40% ormoreandthereforedeliverafasterreadingthanisactuallyachievedbythesamepercentage.The resultofthisvarianceisnotvisibletothetestinguserexceptwhenanapplicationsuchasavideo downloadstartstakinglongertodownloadbecauseavideocannotbecompressedtothesameextentif atall.

TheTimingErrorTest
ThetimingofInternetconnectionspeedrequiresanaccuratetimer,thisgenerallygoeswithoutsaying. Unfortunatelyinamultitasking,timesharingPCenvironmentthisiseasiersaidthandone.Andfaster connectionspeedsdemandhigheraccuracyofmeasurement.Asanexampleofbadtiming,onespeed testingapplicationrecentlyprovidedareportlistingthedatatransferredpersecondwithasummaryof theoverallratepersecondattheendofthereport.However,timingonapersecondbasisisnot accurate.Toclarifythispointfurther,manyyearsagoitwascommontotimeanathleticrunningevent usingapocketstopwatchwhichtimedinseconds.Nowimaginetimingthe2008Olympics400meter freestylewhenMichaelPhelpswonbyjust1000thofasecondusingthesamestopwatch.Theresult wouldhavebeenentirelydifferent,mostprobablyadraw.WhenInternetspeedismeasuredinmillions ofbitspersecond,timingthedatatoatleast1000thofasecondisessentialtomaintainaccuracyofthe result.Thechallengeismaintaininganaccuratetimerto1000thofasecondonaPCplatform.Failureto dothis(andvalidatethatthisisdone)canresultina95Mbpstestreporting85Mbps.

5 2008VisualwareInc.

MakingSenseofBroadbandPerformance

TheRouteVarianceTest
OneofthemostcommonproblemsheardfromISPs,whodealdailywithamultitudeof'speed'related issues,isthattheircustomersuseanyrandomspeedtestertheyfindontheInternetandacceptthe resultswithoutanyregardtolocationrelevanceordistance. Itisimportanttonotethatdistanceisnotmeasuredinmilesbutintime.Justastworoadsthatleadto thesamecitywillbeunlikelytoprovidethesametraveltime,notwoInternetroutestothesame geographicallocationwilldeliverthesameroutetime.Theonlydistancemeasurethatmattersisthe triptime. Wehavealreadydiscussedthatdistancetoadestinationiscriticalinunderstandingandaccepting applicationspeedresults.Thatbeingthecase,whenaspeedtestisrunthatprovidesnofeedbackon triptime,orifthatdistancemeasureisignored,thenthespeedresultismeaningless.Intheoriginal airportcaranalogy,iftheairportwasnot1hourawaybut1minuteaway,thenthe2passengersper hourspeedratingwouldhaveincreasedto120passengersperhour(i.e.4passengersevery2minutes, not4passengersevery2hours). Quiteoftenspeedtestuserswilltesttheirconnectionusingmorethanonespeedtestingapplication thatareinentirelydifferentworldwidelocations,andthenquestionwhytheresultsaredifferent.

ConclusionsSummary
Theconclusionstoderivefromthisdiscussiononspeedtestingarethat: 1. Accuracyisonlyasgoodasthetestingapplicationsabilitytoprovidethedetailsthatvalidate thetestresults. 2. Notallspeedtestersareequalandshouldnotbecomparedunlessthedifferencesaretaken intoconsideration. 3. Theresultsfromaspeedtestershoulddeclarethemethodusedtoconductthetest,otherwise theresultsaremeaningless. 4. Thetestmethodmustaligntotheapplicationprofilefortheresultstobeofvaluewhen assessingtheperformanceexpectationofthatapplication.

6 2008VisualwareInc.