NASA KSC/Intern Report Summer 2015 08/10/15

Augmented/Virtual Reality Laboratory

Student: Benjamin Tully-Hanson

Computer Science/Physics, Junior
University of California, Irvine

Mentor: William Little

Aerospace Technologist, Software Systems
IT-C1
Information Technology and Communication Services Directorate

1Introduction
Realtimemotiontrackinghardwarehasforthemostpartbeencostprohibitivefor
researchtoregularlytakeplaceuntilrecently.WiththereleaseoftheMicrosoftKinectin
November2010,researchersnowhaveaccesstoadevicethatforafewhundreddollarsis
capableofprovidingred/green/blue(RGB),depth,andskeletondata.Itisalsocapableof
trackingmultiplepeopleinrealtime.Foritsoriginalintendedpurposes,i.e.gaming,beingused
withtheXbox360andeventuallyXboxOne,itperformsquitewell.However,researcherssoon
foundthatalthoughthesensorisversatile,ithaslimitationsinrealworldapplications.Iwas
broughtaboardthissummerbyWilliamLittleintheAugmented/VirtualReality(AVR)Labat
KennedySpaceCentertofindsolutionstotheselimitations.
2Setup
WhenmotiontrackingresearchbeganintheAVRLab,itbecameapparentthatusing
multipleKinectsatatimewouldbebeneficial.AttemptsweremadetoattachmultipleKinectsto
asinglecomputer,butduetooversaturationofUSBbandwidththisapproachsimplydidnot
work.Instead,aclient
serverarchitecturewasimplemented.Four
Kinectsarepositionedinasquarefacinginward,eachconnectedto
itsowncomputerdesignatedasaclientmachine.Thepositioning
canbeseeninFigure1.Thesefourmachinesthentransmittheir
separateKinectdataoverthenetworktoaserver,whichthen
processesthecollectionofstreamsandproducesasingleresolved
skeletonthattrackstheuser.
3Limitations
TherearethreemainproblemsthatarisewhenresearchingtheMicrosoftKinects
shortcomings:self
occlusion,bone
lengthvariation,andartificialjointvibration.Asageneral
rule,self
occlusionoccurswhenajointonthehumanbodyishiddenfromtheKinectsviewbya
differentpartofthebody.CommonexamplesincludeauserfacingtheKinectandputtingahand
behindhisorherback,orturning90fromtheKinectandthereforehidingeithertheleftorright
sideofhisorherbody.Inthesescenarios,thejointsthatarehiddenfromthesensorchangefrom

B. Tully-Hanson, Summer 2015 Intern, NASA Kennedy Space Center

Pg. 1

havingatrackedstatetohavinganinferredstate.Whatthismeansisthatalthoughthesensor
doesntknowthepositionofthejoint,itwillmakeaneducatedguessbasedontheanatomyof
thehumanbody.Unfortunately,thisguesscanbeextremelyinaccurateincertainpositions.
Bone
lengthvariationoccurswhenthedistancebetweentwoconnectedjointsonthe
humanbodychanges.Intherealworld,thedistancebetweenanynumberofconnectedjoints
shouldremainconstant.Thisproblemisespeciallyapparentwhentheusermakesaswiftmotion,
suchaskickingoutalegorswinginganarmasiftoserveatennisball.Varioustestshaveshown
bone
lengthtoalmostdoubleincertainscenarios,certainlysomethingthatneedstobefixed.
Artificialjointvibrationisslightlyhardertopinpoint,andcanbeseenevenwhentheuser
isstandingstill.WhentheKinectslightlymistracksajointrepeatedly,thepositioncanoscillate
wildlybetweentwoormorepositions.Thisoscillationisespeciallyapparentinthejointsofthe
usershands.Evenwhenstandingstillwiththehandsawayfromthebodyinanattempttoavoid
mistracking,thehandpointsjumparoundandfrequentlybendinwaysthatareimpossibleforthe
humanbodytobend.
ItistheAVRLabsgoaltobeabletouseasystemofKinectsasarobustandaffordable
methodofmotiontracking.Anumberofapplicationsarepossible,includingrealistictrainingof
astronauts,remotecontrolofhazardousgroundoperations,andmoreefficientrelayof
informationinvideoconferencecalls.However,theabovelistedlimitationsmustbeovercome
fortheKinecttobecomeausefultoolinrealworlduse.
4JointOcclusion
WhenIstartedworkingwiththeKinectsthissummer,themostglaringissuetome
initiallywasthejointocclusion.Iftheusercameclosetoa90angletothesensor,thedisplayed
skeletonbarelylookedhuman.Iwasgivenpaperstoreadconcerningthisissue,themostnotable
being
ImprovedSkeletonTrackingbyDuplexKinects
[1].Thisparticularpaperimplementeda
duplex
Kinectsystem,withtwoKinectsensorsatorthogonalanglestoeachother,inanattempt
toresolvethejointocclusionissue.BycombiningthedatafromKinectswithdifferentviewsof
theperson,thehopeistohavearesolvedskeletonthatismoreaccuratelytracked.Iread
throughthispaperoverthecourseofafewweeks,implementingbitsandpiecesasIwent,and
eventuallycametohaveanintuitiveunderstandingofthealgorithmsoutlined.Figure2shows

B. Tully-Hanson, Summer 2015 Intern, NASA Kennedy Space Center

Pg. 2

anexampleoftherawdata
givenbytheKinectsensor,
alongwithanexamplewithmy
implementedalgorithm.
Thegeneralideabehind
thisalgorithmworksoffofthe
conceptofperspectivedistance
differencebetweentwo
orthogonalviews.Sayfor
examplewehaveaperson
positionedfacingKinectA,as
showninFigure3.KinectA
willhaveaviewofthescene
thatlookssimilartotheimage
inFigure4.WhileKinectB,
whichisata90anglefrom
KinectA,willhaveaviewof
thescenethatlooksverysimilar
totheRawimageinFigure2.
Wefirstcyclethrough
thejointsforeachrespectiveKinect,andfindthejoints
withtheclosestdistancesthatarenotconnectedtoeach
otherbyabone.WhicheverKinecthasthe
smallest
distance,wethenassumethatthosejointscouldhave
beenincorrectlytrackedbytheKinectsensor.Nowthat
wehavethepotentiallyincorrectjoints,wecomparethe
jointdistancesinbothoftheviewsofAandB.
WhicheverKinecthasthe
greater
distance,weassumeto
bemoreaccurateasitislesslikelythatthejointswere

B. Tully-Hanson, Summer 2015 Intern, NASA Kennedy Space Center

Pg. 3

trackedincorrectly.Followingtheaboveprocessusingboththeleftandrighthandjointsas
examples,onecanseethatthealgorithmwouldthenchoosethehandpositionsofKinectAas
beingmoreaccurate.Thissameprocesscarriesthroughouttherestoftheskeleton,andtheresult
isaresolvedskeleton.
5BonelengthVariation
Afterresolvingmostissueswiththejointocclusion,Ibegantonoticemoreclearlythe
changesinbonelength.Thehandsandfeetwouldrepeatedlystretchintheirlength,andcould
evenbeseendoublingatsomepoints.ThegeneralapproachItookwastoimplementa
calibrationstageatthebeginningofthemotiontracking.Inthisstagetheuserwouldstandstill
forabouttwosecondswhileanalgorithmcalculatedthedistancebetweenallthestationary
joints.Aftercompletingthecalculation,thesebonelengthswerethenimplementedashard
constantsforthejointdistances.Ateachframe,aunitvectoriscalculatedthatrepresentsthe
normalizeddirectionofthebone.Fromhereallwemustdoismultiplyitbyitsrespectivebone
length,propagatingthischangedownthelimbs,generatingaconstantbone
lengthskeleton.To
monitortheimpactofmymethod,Iconstructedatablethatmonitorsthevalueofeachbone
length.Whenalengthchangesmorethanadesignatedamountthatrectangleflashesred.Figure
5demonstratestherawdatafromtheKinectwithoutmybone
lengthconstraint.

Asshown,thebonelengthinthisexamplechangesmorethan50%oftheactualvalue.If
thebone
lengthconstraintisenabled,theresultingtrackingismuchmoreaccurate.As
Figure6shows,thebonelengthisconstantwithlessthan0.00001%error.
B. Tully-Hanson, Summer 2015 Intern, NASA Kennedy Space Center

Pg. 4

6ArtificialJointVibration
JointVibration,orwhatwecalljitter,isoneofthemoredifficultchallengestoovercome
withrawKinectdata.Thereareanumberofdampeningalgorithmsthatcanbeimplementedbut
wewalkafinelineofoverdampeningthedataandintroducinglatencyintothesystem,orunder
dampeningandstillhavingsignificantjitter.Althoughthisisstillanareaofactivedevelopment,
myhopeistoeventuallyusethedepthdataprovidedbytheKinecttoreducejitter.Depthdatais
inherentlymorestablethanskeletondata,asinsteadofjusthavingasingledatapointforajoint
wehavealargesurfaceareaofpixelsuponwhichwecanmakecalculations.Bymonitoringthe
numberofdepthpixelswithinacertainareaperframe,wecanusethisasasecondcheckto
determineifajointhastrulymovedornot.
DepthdataisinitiallyreportedbytheKinectasasinglevalue
per
pixelthatsignifiesthe
distanceinmillimetersfromthesensor.Byfilteringthisdata,Iremovethebackgroundandleave
onlyinformationabouttheuser,achievingagreenscreenlikeeffect.IthenassignanRGBvalue
toeachpixeldependingonthereporteddistancetheclosertothecamera,thedarkerthecolor.
Figure7showsanexampleoftheresultingdepthimage.
7DirectionalTracking
Althoughimplementingamulti
Kinectsystemisbeneficialinprovidingmoredatapoints
foraccuratetracking,italsointroducesalimitationofitsown.Kinectsensorswerebuiltwiththe
ideathattheuserwouldbefacingthemdirectly.Assuchtheycannotdistinguishiftheyare

B. Tully-Hanson, Summer 2015 Intern, NASA Kennedy Space Center

Pg. 5

lookingattheusersfrontorback,andwillalways
assumetheformer.Variousresearchpapershave
madementionofbeingabletouseamarkeronthe
frontoftheusertodistinguishittotheKinectsas
thefront.Ihadyettoseeonethattestedand
implementedthismethodeffectively,soIdecided
todoitmyself.
EachKinectclientissetuptopayspecial
attentiontothemiddlespinepositionontheuser,
whilestilltransmittingthefullbody/camera/depth
datatotheserver.Itthenpullsasquaresample
surroundingthispositionfromthecolorstream,and
beginstocheckthecolorofeachpixelwhilealso
displayingitonthescreen.Iftheuserisfacingthe
Kinect,thesampledareashouldbethecolorofthe
marker(inourcaseIusedaredmarker),anda
notificationrectanglewillturngreenasshownin
Figure8.IfinsteadtheuserisnotfacingtheKinect,
thesampledareawillgenerallybeacolorotherthan
themarker,andthenotificationrectanglewillturn
redasshowninFigure9.
Thespecificvaluesofthealgorithmare
modifiablebytheuser,butingeneralitcalculates
thepercentageoftheregionthatiswithinarangeof
redshades.Thispercentageisthenreportedbackto
theserver,whereitcandistinguishbasedonthe
percentagesexactlywhichdirectiontheuseris
facing.Thismethodisnotonlyusefulonitsown,
butcanalsobeusedtoaidinmoreaccurate

B. Tully-Hanson, Summer 2015 Intern, NASA Kennedy Space Center

Pg. 6

resolutionofthejointocclusionlimitation.A
simplecompass
likedisplaywasalsocreatedto
showthedirectionoftheuserinrealtime,as
showninFigure3.

8Conclusion
Thispaperdemonstratesmethodsto
resolvecommonissueswiththeMicrosoft
Kinectmotionsensor.Byimplementingthe
stepsshownhere,amultiKinectsystemcanbe
usedasarobustandcosteffectivewaytotrack
humanmotion.Commonissuesthatare
prevalentinsingleKinecttracking,suchas
jointocclusionandbonelengthvariation,have
beenresolvedintheAVRLabssetup.Future
researchwillfocusonwaystoresolvetheartificialjointvibrationseeninrealtimemotion
tracking,aswellasincreasethefidelityofcurrentmotioncapturetechniques.

References
[1]

ImprovedSkeletonTrackingbyDuplexKinects:APracticalApproachforRealTime

Applications,
J.Yeung,T.Kwok,andC.Wang,TheChineseUniversityofHongKong,Journal
OfComputersandInformationDivisionInEngineering,October16,2013.

B. Tully-Hanson, Summer 2015 Intern, NASA Kennedy Space Center

Pg. 7