13 $ecouple Atte#pt to isolate tro$!le5!ottlenecks =o+nstrea# ser*ices &esp? non.essential( -eigh!oring partitions Asynchronicity to #ini#i@e i#pact o, latency on thro$ghp$t
14 $ecouple A*oid #nesia t;n co$plingB asyncCdirty Dse calls only +hen ret$rning data, else cast Make calls +5 ti#eo$ts onlyB no #onitors -on.!locking casts &nos$spend( so#eti#es arge distri!$tion !$,,ers
15 %aralleli(e Work distri!$tionB start +ith genCser*er Spread +ork to #$ltiple +orkersB genC,actory Spread dispatch to #$ltiple procsB genCind$stry Worker select *ia key &,or d!( or F<FA &,or i5o( 9artitioned ser*ices Ds$? 2.32 partitions pg2 addressing 9ri#ary5secondary &$s$? in pairs(
16 %aralleli(e #nesia Mostly asyncCdirty <solate records to 1 node51 process *ia hashing Each ,rag read5+ritten on only 1 node M$ltiple #nesiaCt#B parallel replication strea#s M$ltiple #nesia dirsB parallel i5o d$ring d$#ps M$ltiple #nesia EislandsF &$s$? 2 nodes5isle( 2etter sche#a ops co#pletion 2etter load.ti#e coordination
17 $ecouple A*oid head.o,.line !locking Separate read " +rite G$e$es Separate inter.node G$e$es A*oid !locking +hen single node has pro!le# -ode.to.node #essage ,or+arding #nesia asyncCdirty replication EH$e$erF F<FA +orker dispatch
21 #ptimi(e Slo+ access to #nesia ta!le +ith lots o, ,rags Acco$nt ta!le has 012 ,rags Sparse #apping o*er islands5partitions A,ter adding hosts, thro$ghp$t +ent do+nI Dn$s$ally slo+ record access An a h$nch, looked at etsBin,o&stats( )ash chains >24 &target is 7(? Aops?
22 #ptimi(e #nesia ,rags &cont?( S#all percentage o, hash !$ckets !eing $sed ets $ses average chain length to trigger split 9define .1:6;1<; %xE=======>L 9define ?@A1L?B6;1<; %x========>L C9define ;1<;6?@?2A1L 33((446)>L
'D optimised !ersion of ma#e6hash (normal aseE atomi #e4) D' 9define .1FE6;1<;(term) G ((is6atom(term) E (atom6ta$(atom6!al(term))-Hslot/$"#et/h!al"e) * G - ma#e6hash&(term)) - .1:6;1<;) C ma#e6hash&6init(term8 ;1<;6?@?2A1L)) - .1:6;1<;)
28 $ecouple Meta.cl$stering i#it si@e o, any single cl$ster Allo+ a cl$ster to span long distances +andistB dist.like transport o*er genCtcp Mesh.connected ,$nctional gro$ps o, ser*ers %ransparent ro$ting layer K$st a!o*e pg2 ocal pg2 #e#!ers p$!lished to ,ar.end All #essages are single.hop
32 "learing the mine&ield 6enerally a!le to detect5de,$se scala!ility #ines !e,ore they e;plode E*ents +hich test the syste# World e*ents &esp? soccer( Ser*er ,ail$res &$s$? RAM( -et+ork ,ail$res 2ad so,t+are p$shes
33 "learing the mine&ield -ot al+ays s$ccess,$lB 2522 o$tage 2egan +ith !ack.end ro$ter glitch Mass node disconnect5reconnect Res$lted in a no*el $nsta!le state Dns$ccess,$l in sta!ili@ing cl$ster &esp? pg2( F$ll stop " restart &,irst ti#e in years( Also $nco*ered an o*erly.co$pled s$!syste# Rolling o$t pg2 patch
34 "hallenges =! scaling, esp? MMS oad ti#e &'1M o!Kects5sec( oad ,ail$res &$nreco*era!le !acklog( 2ottlenecked on disk +rite thro$ghp$t &>700M25s( 9atched a selecti*e.recei*e iss$e, !$t #ore to go Real.ti#e cl$ster stat$s " control at scale A !$nch o, csshN +indo+s no longer eno$gh 9o+er.o,.2 partitioning