2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Developer Zone

How to Emulate Persistent Memory on an Intel Architecture

Server
ByThaiLe(Intel)(https://software.intel.com/enus/user/336165),AddedSeptember27,2016 Translate

Share (https://www.facebook.com/sharer/sharer.php?u=https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architect
Tweet (https://twitter.com/intent/tweet?text=How+to+Emulate+Persistent+Memory+on+an+Intel%C2%AE+Architecture+Server%3A&url=https%3A%2F%
Share (https://plus.google.com/share?url=https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server)

Introduction
Thistutorialprovidesamethodforsettinguppersistentmemory(PMEM)emulationusingregulardynamicrandomaccessmemory(DRAM)onan
IntelprocessorusingaLinux*kernelversion4.3orhigher.Thearticlecoversthehardwareconfigurationandwalksyouthroughsettingupthe
software.Afterfollowingthestepsinthisarticle,you'llbereadytotrythePMEMprogrammingexamplesintheNVMLibrary
(http://pmem.io/nvml/)atpmem.io(http://pmem.io).

Why do this?
IfyoureasoftwaredeveloperwhowantstogetstartedearlydevelopingsoftwareorpreparingyourapplicationstohavePMEMawareness,you
canusethisemulationfordevelopmentbeforePMEMhardwareiswidelyavailable.

What is persistent memory?

Traditionalapplicationsorganizetheirdatabetweentwotiers:memoryandstorage.EmergingPMEMtechnologiesintroducesathirdtier.Thistier
canbeaccessedlikevolatilememory,usingprocessorloadandstoreinstructions,butitretainsitscontentsacrosspowerlosslikestorage.
BecausetheemulationusesDRAM,datawillnotberetainedacrosspowercycles.

Hardware and System Requirements

EmulationofpersistentmemoryisbasedonDRAMmemorythatwillbeseenbytheoperatingsystem(OS)asaPersistentMemoryregion.
BecauseitisaDRAMbasedemulationitisveryfast,butwilllosealldatauponpoweringdownthemachine.Thefollowinghardwarewasusedfor
thistutorial:

CPUandChipset IntelXeonprocessorE52699v4processor,2.2GHz

#ofcoresperchip:22(onlyusedsinglecore)
#ofsockets:2
Chipset:IntelC610chipset,QS(B1step)
Systembus:9.6GT/sIntelQuickPathInterconnect

Platform Platform:IntelServerSystemR2000WTproductfamily(codenamedWildcatPass)

BIOS:GRRFSDP1.86B.0271.R00.1510301446ME:V03.01.03.0018.0BMC:1.33.8932
DIMMslots:24
Powersupply:1x1100W

Memory Memorysize:256GB(16X16GB)DDR42133P

Brand/model:Micron*MTA36ASF2G72PZ2GATESIG

Storage Brandandmodel:1TBWesternDigital*(WD1002FAEX)

Operatingsystem CentOS*7.2withkernel4.5.3

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 1/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software
Table1SystemconfigurationusedforthePMEMemulation.

Linux* Kernel
LinuxKernel4.5.3wasusedduringdevelopmentofthistutorial.Supportforpersistentmemorydevicesandemulationhavebeenpresentinthe
kernelsinceversion4.0,howeverakernelnewerthan4.2isrecommendedforeasierconfiguration.TheemulationshouldworkwithanyLinux
distributionabletohandleanofficialkernel.Toconfiguretheproperdriverinstallation,runmakenconfigandenablethedriver.Pertheinstructions
below,Figures1to5showthecorrectsettingfortheNVDIMMSupportintheKernelConfigurationmenu.

1 $makenconfig
2
3 >DeviceDrivers>NVDIMMSupport>
4
5 <M>PMEM<M>BLK<*>BTT

Figure1:Setupdevicedrivers.

Figure2:SetuptheNVDIMMdevice.

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 2/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Figure3:SetupthefilesystemforDirectAccesssupport.

Figure4:SetupforDirectAccess(DAX)support.

Figure5:NVDIMMSupportproperty.

ThekernelwilloffertheseregionstothePMEMdriversotheycanbeusedforpersistentstorage.Figures6and7showthecorrectsettingforthe
processortypeandfeaturesintheKernelConfigurationmenu.

1 $makenconfig
2
3 >Processortypeandfeatures
4
5 <*>SupportnonstandardNVDIMMsandADRprotectedmemory

Figures4and5showtheselectionsintheKernelConfigurationmenu.

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 3/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Figure6:SetuptheprocessortosupportNVDIMMs.

Figure7:EnableNONstandardNVDIMMsandADRprotectedmemory.

Nowyouarereadytobuildyourkernelusingtheinstructionsbelow.

1 $makejX
2
3 WhereXisthenumberofcoresonthemachine

Duringthenewkernelbuildprocess,thereisaperformancebenefittocompilingthenewkernelinparallel.Anexperimentwithonethreadto
multiplethreadsshowsthatthecompilationcanbeupto95percentfasterthanasinglethread.Withthetimesavedusingmultiplethread
compilationforthekernel,thewholenewkernelsetupgoesmuchfaster.Figures8and9showtheCPUutilizationandtheperformancegainchart
forcompilingatdifferentnumbersofthreads.

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 4/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Figure8:Compilingthekernelsources.

Figure9:Performancegainforcompilingthesourceinparallel.

Install the Kernel

1 #makemodules_installinstall

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 5/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Figure10:Installingthekernel.

ReserveamemoryregionbymodifyingkernelcommandlineparameterssoitappearsasapersistentmemorylocationtotheOS.Theregionof
memorytobeusedisfromsstoss+nn.[KMG]referstokilo,mega,giga.

1 memmap=nn[KMG]!ss[KMG]

Forexample,memmap=4G!12Greserves4GBofmemorybetween12thand16thGB.ConfigurationisdonewithinGRUBandvariesbetween
Linuxdistributions.HerearetwoexamplesofaGRUBconfiguration.

Under CentOS 7.0

1 #vi/etc/default/grub
2 GRUB_CMDLINE_LINUX="memmap=nn[KMG]!ss[KMG]"
3 OnBIOSbasedmachines:
4 #grub2mkconfigo/boot/grub2/grub.cfg

Figure11showstheaddedPMEMstatementintheGRUBfile.Figure12showstheinstructionstomaketheGRUBconfiguration.

Figure11:DefinePMEMregionsinthe/etc/default/grubfile.

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 6/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Figure12:Generatethebootconfigurationfilebasesonthegrubtemplate.

Afterthemachinereboots,youshouldbeabletoseetheemulateddeviceas/dev/pmem0pmem3.Tryingtogetreservedmemoryregionsfor
persistentmemoryemulationwillresultinsplitmemoryrangesdefiningpersistent(type12)regionsasshowninFigure13.Ageneral
recommendationwouldbetoeitherusememoryfromthe4GB+range(memmap=nnG!4G)ortocheckthee820memorymapupfrontandfitting
within.Ifyoudontseethedevice,verifythememmapsettingcorrectnessinthegrubfileasshowninFigure9,followedbydmesg(1)analysisas
showninFigure13.Youshouldbeabletoseereservedrangesasshownonthedmesgoutputsnapshot:dmesg.

Figure13:Persistentmemoryregionsarehighlightedas(type12).

You'llseethattherecanbemultiplenonoverlappingregionsreservedasapersistentmemory.Puttingmultiplememmap="...!..."entrieswillresult
inmultipledevicesexposedbythekernelandvisibleas/dev/pmem0,/dev/pmem1,/dev/pmem2,

DAX - Direct Access

TheDAX(directaccess)extensionstothefilesystemcreatesaPMEMawareenvironment.Somedistros,suchasFedora*24andlater,already
haveDAX/PMEMbuiltinasadefault,andhaveNVMLavailableaswell.OnequickwaytochecktoseeifthekernelhasDAXandPMEMbuiltinto
itistogrepthekernelsconfigfilewhichisusuallyprovidedbythedistrounder/boot.Usethecommandbelow:

1 #egrep(DAX|PMEM)/boot/config`unamer`

Theresultshouldbesomethinglike:

1 CONFIG_X86_PMEM_LEGACY_DEVICE=y
2 CONFIG_X86_PMEM_LEGACY=y
3 CONFIG_BLK_DEV_RAM_DAX=y
4 CONFIG_BLK_DEV_PMEM=m
5 CONFIG_FS_DAX=y
6 CONFIG_FS_DAX_PMD=y
7 CONFIG_ARCH_HAS_PMEM_API=y

ToinstallafilesystemwithDAX(availabletodayforext4andxfs):

1 #mkdir/mnt/pmemdir
2 #mkfs.ext4/dev/pmem3
3 #mountodax/dev/pmem3/mnt/pmemdir
4 Nowfilescanbecreatedonthefreshlymountedpartition,andgivenasaninputtoNVMLpools.

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 7/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Figure14:Persistentmemoryblocks.

Figure15:Makingafilesystem.

Itisadditionallyworthmentioningthatyoucanemulatepersistentmemorywithramdisk(i.e.,/dev/shm)orforcePMEMlikebehaviorbysetting
environmentvariablePMEM_IS_PMEM_FORCE=1.Thiswouldeliminateperformancehitcausedbymsync(2).

Conclusion
Bynow,youknowhowtosetupanenvironmentwhereyoucanbuildaPMEMapplicationwithoutactualPMEMhardware.Withtheadditional
coresonanIntelarchitectureserver,youcanquicklybuildanewkernelwithPMEMsupportforyouremulationenvironment.

References
PersistentMemoryProgramming(http://pmem.io/)

Author(s)
ThaiLeisthesoftwareengineerfocusingoncloudcomputingandperformancecomputinganalysisatIntelCorporation.

For more complete information about compiler optimizations, see our Optimization Notice (/en-us/articles/optimization-notice#opt-en).

Categories:Storage(/enus/search/site/field_topic/storage80693/language/en),Server(/enus/search/site/field_platform/server
20798/language/en),Desktop(/enus/search/site/field_hardware/desktop42644/language/en),Linux*(/en
us/search/site/field_operating_system/linux20787/language/en)

Add a Comment Top

(Fortechnicaldiscussionsvisitourdeveloperforums.Forsiteorsoftwareproductissuescontactsupport.)

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 8/9
2/16/2017 How to Emulate Persistent Memory on an Intel Architecture Server | Intel Software

Pleasesignintoaddacomment.Notamember? Join today

Support TermsofUse *Trademarks Privacy Cookies

Lookforuson:
English

https://software.intel.com/en-us/articles/how-to-emulate-persistent-memory-on-an-intel-architecture-server 9/9