Liang Liang Rong Chen Haibo Chen Yubin Xia KwanJong Park Binyu Zang Haibing Guan Institute of Parallel and Distributed Systems Shanghai Jiao Tong University China ID: 619796
Download Presentation The PPT/PDF document "A Case for Virtualizing Persistent Memor..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
A Case for Virtualizing Persistent Memory
Liang Liang, Rong Chen, Haibo Chen, Yubin Xia, KwanJong Park, Binyu Zang, Haibing Guan
Institute of Parallel and Distributed
SystemsShanghai Jiao Tong University, China
Samsung
Electronics Co., Ltd
.Slide2
Tension between Performance
and PersistenceVolatile storagePerformance ✔ vs. Persistence ✘Non-volatile storagePerformance ✘ vs. Persistence ✔Persistent memoryPerformance ✔ vs. Persistence ✔2Low latencyFeaturesNon-Volatile
Performance and PersistenceSlide3
Hardware vendor
Software developer3Data CentersCloud ProvidersPrediction:PM will soon be adopted
Proliferation of PMSlide4
4
≈….Cost EffectivenessPerformanceLoad/Store
fast
slow
Need of PM VirtualizationSlide5
Outline
The Need of PM VirtualizationDesign of VPM.ImplementationEvaluationConclusion5OutlineSlide6
VM apps
Virtualized PMPMSSDDRAM
Hardware
PM
VPM: OverviewSlide7
VM apps
PMSSDDRAM
Hardware
VPM
Virtualized PM
Hypervisor
Guest VM
Emulate Persistent Memory(PM)
with
mix
of
PM, DRAM
and
hard
disk(SSD)
VPM: Overview
Challenge
How to emulate persistency of PM
w
ith other storage medium? Slide8
8
Design: Two ApproachesFull VirtualizationRetain Transparency to guest VMs.Para-VirtualizationEnable guest VMs to provide hints to hypervisorto improve performance.Slide9
9
Full Virtualization: VPM-HVEPTMapping for virtualized PM
PM
RAM
SSD
Read request is issued over PM.
Read request is handled.Slide10
10
Full Virtualization: VPM-HVEPTMapping for virtualized PM
PM
RAM
SSD
Read request is issued over DRAM.
Read request is handled.Slide11
11
Full Virtualization: VPM-HVEPTMapping for virtualized PM
PM
RAM
SSD
Write request is issued over PM.
Write request is handled.Slide12
12
Full Virtualization: VPM-HVEPTMapping for virtualized PM
PM
RAM
SSD
Write request is issued over DRAM.
Trap and remap DRAM with PM.
Write request is handled.Slide13
13
Full Virtualization: VPM-HVEPTMapping for virtualized PM
PM
RAM
SSD
Write over non-PM mapped memory will always trap to hypervisor.Slide14
14
Write Protected (DRAM)Not mappedWritable (NVRAM)On Disk
PMEM
VM0 VPM DS
VM0
DRAM
VM0
PMEM-PT
NVRAM
NVRAM
Shared across all VM
VM0
VP-meta
Mapped
to
EPT
Backing Block n
VM0
Disk
Backing
Block m
…
backed
VM1
VPM DS
Reserved
for
metadata
NVM Pool
DRAM
VPM-HV: Architecture
VPMSlide15
15
Design: Two ApproachesFull VirtualizationRetain Transparency to guest VMs.Para-VirtualizationEnable guest VMs to provide hints to hypervisorto improve performance.Slide16
16
VPM-PV: Four Interfacesvpm_protect(page)vpm_unprotect(page)vpm_persist(range)vpm_pcommit(void)Claim that a page is being modified by guest.Claim that some pages should be persisted.
Wait until all persist requests have been done.Slide17
17
Para-Virtualization: VPM-PVEPT
PM
RAM
SSD
Write request is wanted.
Write request is handled.
Write request is issued over DRAM.
Lock page
Unlock pageSlide18
18
Para-Virtualization: VPM-PVEPT
PM
RAM
SSD
Issue a persist request
Return immediately
Persist
Q
vpm_persistSlide19
19
Para-Virtualization: VPM-PVEPT
PM
RAM
SSD
Persist thread will consume these requests in the background
Do nothing to a PM mapped page
Persist
Q
Persist
ThreadSlide20
20
Para-Virtualization: VPM-PVEPT
PM
RAM
SSD
Persist thread will consume these requests in the background
Do nothing to a PM mapped page
Persist
Q
Persist
Thread
Apply HOT DRAM mapped page to PMSlide21
21
Para-Virtualization: VPM-PVEPT
PM
RAM
SSD
Persist thread will consume these requests in the background
Do nothing to a PM mapped page
Persist
Q
Persist
Thread
Apply HOT DRAM mapped page to PM
Persist COLD DRAM mapped page to SSDSlide22
22
Para-Virtualization: VPM-PVEPT
PM
RAM
SSD
Persist thread will consume these requests in the background
Do nothing to a PM mapped page
Persist
Q
Persist
Thread
Apply HOT DRAM mapped page to PM
Persist COLD DRAM mapped page to SSDSlide23
23
Para-Virtualization: VPM-PVIssue pcommitPersist QPersist Thread
VPM
Block until all requests are handled
Pcommit
is handledSlide24
24
Writable (DRAM)Not mappedWritable (NVRAM)On DiskVPM-PV: Architecture
PMEM
VM0
VPM DS
VM0
DRAM
VM0
PMEM-PT
NVRAM
NVRAM
Shared across all VM
VM0
VP-meta
Backing Block n
Backing
Block m
…
VM1
VPM DS
Reserved
for
metadata
NVM Pool
DRAM
Shared
M
VPM
Shared
M
HypercallSlide25
Outline
The Need of PM VirtualizationDesign of VPM.ImplementationEvaluationConclusion25OutlineSlide26
Guest OS
Kernel module for managing and allocating PMUses reserved consecutive physical memory region, backed by VPMVPM HypervisorTracking of PMVM exit event for VPM-HVUse vpm_persist interface for VPM-PVManaging of PMRemaps the guest PM page to a DRAM page if the updating frequency is not high enough.26VPM ImplementationSlide27
Batching
of flushing to diskSwap Out Disk FlushFlush is not emergent if swapped page is not taken promptlyDirtied data reaches a certain threshold or on applications’ requestReducing writing cost with lazy reallocationDoes not immediately clear if swapped page is not taken promptly Its content and mapping info still retain until it is reused by others27Optimization DecompositionSlide28
Outline
The Need of PM VirtualizationDesign of VPM.ImplementationEvaluationConclusion28OutlineSlide29
PM Types
NV-DIMMPCMVirtualization TypesVPM-HVVPM-PV29PMBD throughput slowdown EvaluationSlide30
30
a. Achieve over 80% of the performance with only 20% of the PMb. PV Outperforms than base line in PCM.Fileserver over NVDIMM/PCMSlide31
31
Frequent flush, introduces many VMExits, which impairs the chance to hide the flush latency in the background.Varmail over NVDIMM/PCMSlide32
32
startcrashcrashInject CrashDrop EPT entriesCrash Recovery (Emulate)Slide33
A study on interfaces and hypervisor to virtualize PM
A prototype: VPMFull-virtualization of PMPara-virtualization of PMEfficiently manage and multiplex PM, leading to small performance degradation33Thankshttp://ipads.se.sjtu.edu.cn/ConclusionSlide34
Tracking of PM
Hardware virtualization of PMLeverages the Access/Dirty Bit on EPT entries, tracks PM pages every 5ms in VM-Exit eventChecking ALL PM pages will hurt performanceTwo-level scanning, pages will be exchanged between 2 levelHot pages for each roundCold pages will be scanned in a clock mannerPara-virtualization of PMObtained directly when processing vpm_persist APIs.Slide35
Managing of PM
Hardware virtualization of PMHandling PM access EPT-violationAllocate native PMIn the worst situation, it revokes an already allocated PM pageGuest PM Page remappingRemaps the guest PM page to a DRAM page if the updating frequency is not high enough.Happens at the same frequency as the PM access tracking.Para-virtualization of PMHandling PM access EPT-violationTry to allocate native PM, but it is not compulsoryPersist Thread in HypervisorShare memory with each guest VM: protect/unprotect, persist requestWhen vpm_pcommit comes, persist thread flushes whole queue35Slide36
Challenge of PM
VirtualizationAbstraction of PMBlock device and memoryPVConsolidate PM to maximize cost-efficiency, performancePut read-most data in DRAMTDPCrash recovery in hybrid memory architectureReserve key data structures in PM36Slide37
Crash Recovery
37Slide38
Using PM
38Data CentersCloud ProvidersIndustry ‘s ReactionLow latencyFeaturesNon-Volatile
PM has growing into a trend.Slide39
39
Application BenchmarkPerformance for different workloads under consolidationFileserver and webserver are hardly effected