vSphere 6.0 – Transparent Page Sharing, Salting, and Memory States

Transparent page sharing (TPS) is a mechanism where the vmkernel automatically identifies identical pages of virtual memory and consolidates them to a single physical memory page, in a manner that is transparent to the virtual machine guest operating system.   In vSphere 6, intra-VM TPS is enabled by default and inter-VM TPS is disabled by default, due to some security concerns as described in VMware KB 2080735. You can change the TPS behavior by applying the salting mechanism as described in VMware KB 2097593. A low priority, scheduled vmkernel process uses a hashing mechanism to apply TPS by identifying matching pages within a virtual machine and other processes, depending on TPS settings. Naturally, the amount of page sharing depends on factors such as the amount of matching pages among the virtual machines and other worlds. The amount of memory pages claimed by inter-VM TPS, when enabled, can rise as the number of virtual machines running the same guest OS rises. This number is negatively impacted by the use of large pages (2 MB pages).   Most modern guest operating systems use large pages, which cannot be shared by the vmkernel.   Page sharing is not a direct sign of memory contention, since it may be engaged regardless of the memory state; however, in the Clear memory state, the TPS service is actively called instead of waiting on the next scheduled TPS run. Also in the High memory state, the ESXi host begins breaking large memory pages into small pages, which facilitates an increase in page sharing.

In vSphere 5.x, TPS could be considered as an indirect sign of memory contention, since it tends to only engage significantly when the ESXi host is experiencing memory pressure. Per VMware KB 2080735, VMware disabled inter-VM page sharing by default in updates for VMware 5.x.

You can set a virtual machine’s advanced parameter sched.mem.pshare.salt to control its ability to participate in transparent page sharing.   TPS is only allowed within a virtual machine (intra-VM TPS) by default, because the ESXi host configuration option Mem.ShareForceSalting is set to 2, the sched.mem.pshare.salt is not present in the virtual machine configuration file, and thus the virtual machine salt value is set to unique value.   In this case, to allow TPS among a specific set of virtual machines, set the sched.mem.pshare.salt of each virtual machine in the set to an identical value.  Alternatively, to enable TPS among all virtual machines (inter-VM TPS), you can set Mem.ShareForceSalting to 0, which causes sched.mem.pshare.salt to be ignored and to have no impact. Or, to enable inter-VM TPS as the default, but yet allow the use of sched.mem.pshare.salt to control the effect of TPS per virtual machine, set the value of Mem.ShareForceSalting to 1. In this case, change the value of sched.mem.pshare.salt per virtual machine to prevent it from sharing with all virtual machines and restrict it to sharing with those that have an identical setting.

in ESXi 6.0, the five memory states are High, Clear, Soft, Hard, and Low, where Clear is a new state in version 6.0. The states are based on a value called minFree, which is 899 MB for the first 28 GB of ESXi host memory plus 1% of any additional host memory. For example, for a 100GB ESXi host, minFree is 899 MB plus 1% of (100 GB – 28GB) = 1619 MB. The thresholds for each state is based on the amount of free available memory in the host compared to the minFree value. The thresholds are High equals 400% minFree, Clear equals 100% minFree, Soft equals 64% minFree, Hard equals 32%, and Low equals 16%. When the free ESXi host memory drops below 400% minFree (the High threshold), ESXi begins breaking large pages into small pages. At the Clear threshold, TPS is actively called, instead of waiting for next TPS run. At the Soft threshold, ballooning begins. At the Hard threshold, compression and swapping begins. At the Low threshold, blocking begins, where certain virtual machines are prevented from allocating memory.

The main impact of the Clear threshold is that you should expect to see increased page sharing activity well before you see ballooning.  Ballooning will not engage until the Soft threshold (64%) is crossed, but page sharing may engage at any time, and will actively engage as the Clear threshold (100%) is crossed.

In the vSphere Web Client performance graphs, the Memory – Shared metric can be used to monitor the size in kilobytes (KB) of the amount of memory that is shared by TPS, which reduces the need to engage memory ballooning, compression or swapping.