Memory Resource Management in VMware ESX Server - PowerPoint Presentation

Slide1

Memory Resource Management in VMware ESX Server

By Carl A. WaldspurgerPresented by Clyde Byrd III(some slides adapted from C. Waldspurger)

EECS 582 – W16

1Slide2

Overview

BackgroundMemory ReclamationPage SharingMemory Allocation PoliciesRelated WorkConclusionEECS 582 – W16

2Slide3

Background

Server consolidation neededMany physical servers underutilizedConsolidate multiple workloads per machine Using Virtual Machines

VMware ESX Server: a thin software layer designed to multiplex hardware resources efficiently among virtual machines

Virtualizes the Intel IA-32 architecture

Runs existing operating systems without modification

EECS 582 – W16

3Slide4

ESX Server

EECS 582 – W164Slide5

Memory Abstractions in ESX

TerminologyMachine address: actual hardware memory“Physical” address: a software abstraction used to provide the illusion of hard ware memory to a virtual machinePmap

: for each VM to translate “physical” page numbers (PPN) to machine page numbers (MPN)

Shadow page tables: contain virtual-to-machine page mappings

EECS 582 – W16

5Slide6

Memory Abstractions in ESX (cont.)

EECS 582 – W166Slide7

Memory Reclamation

Traditional approachIntroduce another level of paging, moving some VM “physical” pages to a swap area on diskDisadvantages

:

Requires a meta-level page replacement

policy

Introduces performance

Double paging problem

EECS 582 – W16

7Slide8

Memory Reclamation (cont.)

The hip approach: Implicit cooperationCoax guest into doing the page replacementAvoid meta-level policy decisionsEECS 582 – W168Slide9

Ballooning

EECS 582 – W169Slide10

Ballooning (cont.)

EECS 582 – W1610

The

black bars plot the performance when the VM is configured with main memory sizes ranging from 128 MB to 256

MB

The

gray bars plot the performance of the same VM configured with 256 MB, ballooned down to the specified

size

Throughput of a Linux VM running

dbench

with 40

clients Slide11

Ballooning (cont.)

Ballooning is not available all the time: OS boot time, driver explicitly disabledBallooning does not respond fast enough for certain situationsGuest OS might have limitations to upper bound on balloon sizeEECS 582 – W1611Slide12

Page Sharing

ESX Server exploits redundancy of data across VMsMultiple instances of the same guest OS can and do share some of the same data and applicationsSharing across VMs can reduce total memory usageThe system allows tries to share a page before swapping out pagesEECS 582 – W1612Slide13

Page Sharing (cont.)

Content-Based Page SharingIdentify page copies by their contents. Pages with identical contents can be shared regardless of when, where or how those contents were generatedBackground activity saves memory over time

Advantages:

Eliminates the need to modify, hook or even understanding guest OS

code

Able to identify more opportunities for

sharing

EECS 582 – W16

13Slide14

Page Sharing (cont.)

EECS 582 – W1614Scan Candidate PPNSlide15

Page Sharing (cont.)

EECS 582 – W1615

Successful MatchSlide16

Page Sharing (cont.)

EECS 582 – W1616Slide17

Memory Allocation policies

ESX allows proportional memory allocation for VMsWith maintained memory performanceWith VM isolationAdmin configurable { min, max, shares }EECS 582 – W1617Slide18

Proportional allocation

Resource rights are distributed to clients through TICKETSClients with more tickets get more resources relative to the total resources in the systemIn overloaded situations client allocation degrades gracefullyProportional-share can be unfair, ESX uses an “idle memory tax” to be more reasonableEECS 582 – W1618Slide19

Idle Memory Tax

Tax on idle memoryCharge more for idle page than active pageIdle-adjusted shares-per-page ratioTax rateExplicit administrative parameter0% is too unfair, 100% is too aggressive, 75% is the defaultHigh default rateReclaim most idle memorySome buffer against rapid working-set increasesEECS 582 – W16

19Slide20

Idle Memory Tax (cont.)

The tax rate specifies the max number of idle pages that can be reallocated to active clientsWhen an idle paging client starts increasing its activity the pages can be reallocated back to full shareESX statistically

samples pages in each VM to estimate active memory usage

ESX

by default samples 100 pages every 30 seconds

EECS 582 – W16

20Slide21

Idle Memory Tax (cont.)

EECS 582 – W1621

Experiment:

2 VMs, 256 MB, same shares.

VM1

: Windows

boot+idle

.

VM2

:Linux

boot+dbench

.

Solid: usage,

Dotted:active

.

Change tax rate 0%

 75%

After: high tax.

Redistribute

VM1

→

VM2

.

VM1

reduced to min size.

VM2

throughput improves 30%Slide22

Dynamic Reallocation

ESX uses thresholds to dynamically allocate memory to VMsESX has 4 levels from high, soft, hard and lowThe default levels are 6%, 4%, 2% and 1%ESX can block a VM above target allocations when levels are at

lowRapid state fluctuations are prevented by changing back to higher level only after higher threshold is significantly exceeded

EECS 582 – W16

22Slide23

Dynamic Reallocation

EECS 582 – W1623Slide24

I/O Page Remapping

IA-32 supports PAE to address up to 64GB of memory over a 36bit address spaceESX can remap “hot” pages in high “

physical” memory addresses to lower machine addresses

EECS 582 – W16

24Slide25

Related Work

Disco and Cellular DiscoVirtualized servers to run multiple instances of IRIXVmware WorkstationType 2 hypervisor; ESX is type 1

Self-paging of the Nemesis system

Similar to Ballooning

Requires applications to handle their own virtual memory operations

Transparent page sharing work in Disco

IBM’s MXT memory compression technology

Hardware

approach; but can be achieved through page sharing

EECS 582 – W16

25Slide26

Conclusion

Key featuresFlexible dynamic partitioningEfficient support for overcommitted workloads Novel mechanismsBallooning leverages guest OS algorithms Content-based page sharingIntegrated policiesProportional-sharing with idle memory

taxDynamic reallocation

EECS 582 – W16

26