4/30/2014

Motivation

EndtoendDataIntegrity
forFileSystems

Filesystemscachea largeamountofdatainmemory
forperformance

Memorycapacityisgrowing

FromZFStoZ2FS

Filesystemsmaycachedataforalongtime
Susceptibletomemorycorruption

YupuZhang
yupu@cs.wisc.edu

Howrobustaremodernfilesystemstomemory
corruption?
AZFScasestudy
4/30/2014

Fletcher

ZFSBackground
write()

4/30/2014

ReliabilityAnalysisofZFS

read()

Questions
Whathappenswhenmemory corruptionoccurs?
Howlikelyabitflipwouldcauseproblems?

MEM

Method

DISK

Randomtest:injectbitflipsrandomly
Controlledtest:targetZFSmetadataanddata
t0

t1
Generate

t2

t3

Results

Verify

Read/writecorruptdata,systemcrash,operationfailures
Onebitfliphasnonnegligiblechance(ashighas7%)of
causingfailures

Ondiskblocksarewellprotected
Howabout blocksinmemory?
4/30/2014

4/30/2014

4/30/2014

PreviousStateoftheArt

TwoDrawbacks
Performance

EndtoendDataIntegrity

Repeatedlyaccessingdatafrominmemorycache
Strongchecksummeanshighoverhead

Checksumforeachdatablockisgeneratedandverified
byapplication
Same checksumprotectsdatathroughoutentirestack
Astrong checksumisusuallypreferred

Timeliness
Itistoolatetorecoverfromthecorruptionthatoccursbeforea
blockiswrittentodisk
Verify
Checksum

Generate
Checksum

FAIL

WritePath

ReadPath

4/30/2014

WritePath
5

4/30/2014

unbounded
time

ReadPath
6

ZettabytereliableZFS(Z2FS)

Flexible EndtoendDataIntegrity

Fletcher(slow,butstrong)
xor (fast,butweak)

write()

Cooperation fromapplication,pagecache,andFS

read()

MEM

Changechecksumacrosscomponentsorovertime

Performance

DISK

Fastbutweakerchecksumforinmemorydata
Slowbutstrongerchecksumforondiskdata

t0
Generate

Timeliness
Eachcomponentisawareofthechecksum
Verificationcancatchcorruptionintime

t1
Generate
Verify

t2
Verify
Generate

ChecksumChaining
Coverageoverlaps
Detectcorruptionintime

4/30/2014

4/30/2014

Verify

t3
Verify

vulnerablewindow!
Fletcherasdiskchecksum
xor asmemorychecksum
Betterperformance
8

4/30/2014

ZettabytereliableZFS(Z2FS)

Fletcher(slow,butstrong)

ReliabilityAnalysis

xor (fast,butweak)

write()

read()

Ananalytical framework

MEM

Modeldevices(memory,disk)andchecksums
Exploreawiderangeofreliabilityparameters
Best
Best

DISK

t0

t1

t2

t3

Consumer

Consumer
Server

Generate

Generate
Verify

Verify

ChecksumChaining

Verify
Verify

Worst

Server
Worst

Reasonaboutwhen,wheretousewhatchecksum
ShowthatZ2FSisabletoprovideZettabyte Reliability

ChecksumSwitching

Longerresidencytime=>datamorelikelybeingcorrupt
4/30/2014

4/30/2014

10

Fletcher

ReliabilityTest

PerformanceEvaluation

xor

write()

Micro&MacroBenchmark
NormalizedThroughput

MEM

DISK

t0

t1

Generate

Generate
Verify
FAIL

EndtoendZFS(Fletcher)

ZFS(static)

ZFS(dynamic)

1
0.8
0.6
0.4
0.2
0
SeqRead(cold)

SeqRead(warm)

reada1GBfile

Asktheapplicationtorewrite

webserver
WarmReadintensive

varmail
Dominately by
RandomI/Os

Betterprotectionusuallymeanshigheroverhead
Z2FShelpstoreducetheoverhead,especiallyforwarmreads

Z2FSdetectsandrecoversfromcorruptionintime
4/30/2014

ZFS

11

4/30/2014

12

4/30/2014

Summary
Endtoenddataintegrityhelpsbutisnotperfect
Slowperformance,anduntimelydetection
Onechecksumdoesnotalwaysfitall

Solution: flexibleendtoenddataintegrity
Cooperation amongcomponents
Choosethebestchecksumbasedondeviceproperties

ImplementationofZ2FS
Checksumchaining+Checksumswitching
LOC: ~6000inZFS
4/30/2014

13