{"id":35292,"date":"2019-10-31T22:03:28","date_gmt":"2019-10-31T19:03:28","guid":{"rendered":"https:\/\/prohoster.info\/blog\/analiz-proizvoditelnosti-vm-v-vmware-vsphere-chast-2-memory\/"},"modified":"2019-10-31T22:03:28","modified_gmt":"2019-10-31T19:03:28","slug":"analiz-proizvoditelnosti-vm-v-vmware-vsphere-chast-2-memory","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/analiz-proizvoditelnosti-vm-v-vmware-vsphere-chast-2-memory","title":{"rendered":"Performance analysis of VMs in VMware vSphere. Part 2: Memory","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Performance
\n
\nPart 1. About CPU<\/a><\/noindex><\/p>\n

This article discusses performance counters for RAM in vSphere.
\nMemory issues seem to be more straightforward than CPU ones: if a VM encounters performance problems, they are hard to miss. However, once they appear, it's much more challenging to address them. But let's take it step by step. <\/a><\/noindex><\/p>\n

A Bit of Theory<\/h3>\n

\nThe RAM for virtual machines is drawn from the memory of the servers on which the VMs operate. This is quite obvious :) If the server's RAM is insufficient for all requests, ESXi begins to apply memory reclamation techniques. Otherwise, the operating systems of the VMs would crash with access errors to RAM. <\/p>\n

The techniques ESXi applies depend on the memory load:<\/p>\n

Memory State<\/b><\/p>\n

Threshold<\/b><\/p>\n

Actions<\/b><\/p>\n

High<\/p>\n

400% of minFree<\/p>\n

Once the upper threshold is reached, large memory pages are broken down into smaller ones (TPS operates in standard mode).<\/p>\n

Clear<\/p>\n

100% of minFree<\/p>\n

Large memory pages are broken down into smaller ones, TPS operates forcefully.<\/p>\n

, which is located within the driver\u2019s codebase itself. It uses its own<\/p>\n

64% of minFree<\/p>\n

TPS + Balloon<\/p>\n

Hard<\/p>\n

32% of minFree<\/p>\n

TPS + Compress + Swap<\/p>\n

Low<\/p>\n

16% of minFree<\/p>\n

Compress + Swap + Block<\/p>\n

Source<\/a><\/noindex> <\/p>\n

minFree is the amount of RAM necessary for the hypervisor to operate. <\/p>\n

Until ESXi 4.1, minFree was by default fixed at 6% of the server's RAM (the percentage could be changed using the Mem.MinFreePct option in ESXi). In later versions, due to the increasing amounts of memory on servers, minFree began to be calculated based on the host's memory size rather than as a fixed percentage. <\/p>\n

The default minFree value is calculated as follows:<\/p>\n

Percentage of memory reserved for minFree<\/b><\/p>\n

Memory range<\/b><\/p>\n

6%<\/p>\n

0-4 GB<\/p>\n

4%<\/p>\n

4-12 GB<\/p>\n

2%<\/p>\n

12-28 GB<\/p>\n

1%<\/p>\n

Remaining memory<\/p>\n

Source<\/a><\/noindex><\/p>\n

For example, for a server with 128 GB of RAM, the MinFree value would be as follows:
\nMinFree = 245.76 + 327.68 + 327.68 + 1024 = 1925.12 MB = 1.88 GB
\nThe actual value may differ by a few hundred MB, depending on the server and memory.<\/p>\n

Percentage of memory reserved for minFree<\/b><\/p>\n

Memory range<\/b><\/p>\n

Value for 128 GB<\/b><\/p>\n

6%<\/p>\n

0-4 GB<\/p>\n

245.76 MB<\/p>\n

4%<\/p>\n

4-12 GB<\/p>\n

327.68 MB<\/p>\n

2%<\/p>\n

12-28 GB<\/p>\n

327.68 MB<\/p>\n

1%<\/p>\n

Remaining memory (100 GB)<\/p>\n

1024 MB<\/p>\n

\nTypically, the High state is considered normal for production environments. For testing and development environments, Clear\/Soft states may be acceptable. If less than 64% of MinFree memory remains on the host, VMs running on it will undoubtedly experience performance issues.<\/p>\n

Specific memory reclamation techniques are applied in each state, starting with TPS, which has a minimal impact on VM performance, up to swapping. I'll explain them in more detail. <\/p>\n

Transparent Page Sharing (TPS).<\/b> TPS is, roughly speaking, the deduplication of virtual machine memory pages on the server.<\/p>\n

ESXi looks for identical virtual machine memory pages by calculating and comparing the hash values of the pages and removes duplicates, replacing them with references to the same page in the server's physical memory. As a result, physical memory consumption decreases, allowing for some memory oversubscription without significantly impacting performance.<\/p>\n

\"Performance
\nSource<\/a><\/noindex><\/p>\n

This mechanism only works for memory pages sized at 4 KB (small pages). The hypervisor doesn't even attempt to deduplicate 2 MB (large pages) as the chances of finding identical pages of that size are low.<\/p>\n

By default, ESXi allocates memory for large pages. Fragmentation of large pages into small ones begins when the high state threshold is reached and occurs forcefully when the clear state is achieved (see the hypervisor state table).<\/p>\n

If you want TPS to start working without waiting for the host's memory to fill up, you need to set the value in the Advanced Options of ESXi to \u201cMem.AllocGuestLargePage\u201d<\/i> to 0 (default is 1). Then, the allocation of large memory pages for virtual machines will be disabled.<\/p>\n

Since December 2014, TPS between VMs has been disabled by default in all ESXi releases due to the discovery of a vulnerability that theoretically allows one VM to access the memory of another VM. Details can be found here. I have not encountered practical implementation information about exploiting the TPS vulnerability.<\/p>\n

The TPS policy is controlled through the advanced option \u201cMem.ShareForceSalting\u201d<\/i> on ESXi:
\n0 \u2014 Inter-VM TPS. TPS works for pages of different VMs;
\n1 \u2013 TPS for VMs with the same value of \u201csched.mem.pshare.salt\u201d in VMX;
\n2 (default) \u2013 Intra-VM TPS. TPS works for pages within the VM.<\/p>\n

It definitely makes sense to disable large pages and enable Inter-VM TPS on test environments. This can also be useful for environments with a large number of similar VMs. For example, in VDI environments, the savings in physical memory can reach tens of percent. <\/p>\n

Memory Ballooning.<\/b> Ballooning is no longer as harmless and transparent for the VM operating system as TPS. However, with proper application, one can live and even work with Ballooning.<\/p>\n

Along with VMware Tools, a special driver called the Balloon Driver (also known as vmmemctl) is installed on the VM. When the hypervisor runs low on physical memory and transitions to a Soft state, ESXi asks the VM to return unused RAM through this Balloon Driver. The driver, in turn, operates at the level of the operating system and requests free memory from it. The hypervisor sees which pages of physical memory the Balloon Driver occupies, takes memory from the virtual machine, and returns it to the host. There are no issues with the OS's operation since the memory is occupied by the Balloon Driver at the OS level. By default, the Balloon Driver can reclaim up to 65% of the VM's memory.<\/p>\n

If VMware Tools are not installed on the VM or Ballooning is disabled (not recommended, but there are situations), the hypervisor immediately resorts to more severe memory reclamation techniques. KB<\/a><\/noindex>:), the hypervisor immediately moves to stricter memory reclamation techniques. Conclusion: ensure VMware Tools are installed on the VM.<\/p>\n

\"Performance
\nThe operation of the Balloon Driver can be checked from the OS through VMware Tools.<\/i>.<\/p>\n

Memory Compression.<\/b> This technique is applied when ESXi reaches a Hard state. As the name suggests, ESXi attempts to compress 4 KB memory pages to 2 KB in order to free up a bit of space in the server's physical memory. This technique significantly increases the access time to the contents of the VM's memory pages, as the page needs to be decompressed first. Sometimes not all pages can be compressed, and the process itself takes some time. Therefore, this technique is not very effective in practice.<\/p>\n

Memory Swapping.<\/b> After a brief phase of Memory Compression, ESXi almost inevitably (unless VMs have migrated to other hosts or powered off) transitions to Swapping. If there is very little memory left (Low state), the hypervisor also stops allocating memory pages to the VMs, which may cause problems in the guest operating systems of the VMs.<\/p>\n

This is how Swapping works. When a virtual machine is powered on, a file with the .vswp extension is created for it. Its size equals the unreserved RAM of the VM: the difference between the configured and reserved memory. During Swapping, ESXi unloads the memory pages of the virtual machine into this file and begins to use it instead of the physical memory of the server. Naturally, this type of \"operational\" memory is several orders of magnitude slower than actual memory, even if the .vswp file is on fast storage.<\/p>\n

Unlike Ballooning, where unused pages are reclaimed from the VM, during Swapping, pages that are actively used by the OS or applications within the VM may be written to disk. As a result, the VM's performance may drop to the point of stalling. The VM technically continues to operate, and it can at least be properly shut down from the OS. If you are patient \ud83d\ude09<\/p>\n

If the VMs have gone into Swap, this is an abnormal situation that should be avoided whenever possible.<\/p>\n

Key memory performance counters of the virtual machine<\/h3>\n

\nNow we have reached the main point. The following counters are available for monitoring memory states in VMs:<\/p>\n

Active<\/b> \u2014 shows the amount of memory (KB) that the VM had access to in the previous measurement period.<\/p>\n

Usage<\/b> \u2014 the same as Active, but expressed as a percentage of the VM's configured RAM. Calculated using the following formula: active \u00f7 virtual machine configured memory size.
\nHigh Usage and Active, respectively, are not always indicators of VM performance issues. If the VM is aggressively using memory (at least accessing it), it does not mean that there is not enough memory. Rather, this is a reason to check what is happening in the OS.
\nThere is a standard Alarm for Memory Usage for VMs:<\/p>\n

\"Performance
\n
\nShared<\/b> \u2014 the amount of memory used by the VM that has been deduplicated using TPS (within the VM or between VMs).<\/p>\n

Granted<\/b> \u2014 the amount of physical memory of the host (KB) that has been allocated to the VM. Includes Shared.<\/p>\n

Consumed<\/b> (Granted \u2014 Shared) \u2014 the amount of physical memory (KB) that the VM consumes from the host. Does not include Shared.<\/p>\n

If some memory of the VM is not drawn from the host's physical memory but from the swap file or is reclaimed from the VM via Balloon Driver, this amount is not included in Granted and Consumed.
\nHigh values for Granted and Consumed are completely normal. The operating system gradually takes memory from the hypervisor and does not give it back. Over time, for an actively running VM, the values of these counters approach the size of the configured memory and stay there.<\/p>\n

Zero<\/b> \u2014 the amount of memory for the VM (KB) that contains zeros. This memory is considered free by the hypervisor and can be given to other virtual machines. Once the guest OS writes anything to the zeroed memory, it transitions to Consumed and does not return.<\/p>\n

Reserved Overhead<\/b> \u2014 the amount of memory for the VM (KB) reserved by the hypervisor for the VM's operation. This is a small amount, but it must be available on the host; otherwise, the VM will not start.<\/p>\n

Balloon<\/b> \u2014 the amount of memory (KB) taken from the VM using the Balloon Driver.<\/p>\n

Compressed<\/b> \u2014 the amount of memory (KB) that was successfully compressed.<\/p>\n

Swapped<\/b> \u2014 the amount of memory (KB) that, due to the lack of physical memory on the server, has moved to disk.
\nBalloon and other memory reclamation technique counters are zero.<\/p>\n

This is what the graph with the counters for a normally functioning VM with 150 GB of RAM looks like.<\/p>\n

\"Performance
\n
\nIn the graph below, the VM has clear problems. Below the graph, it can be seen that all the described techniques for managing memory have been used for this VM. The Balloon for this VM is significantly larger than Consumed. In fact, the VM is more likely dead than alive. <\/p>\n

\"Performance
\n<\/p>\n

ESXTOP<\/h3>\n

\nAs with the CPU, if we want to quickly assess the situation on the host and its dynamics with intervals up to 2 seconds, we should use ESXTOP.<\/p>\n

The ESXTOP screen for Memory is invoked by pressing the key 'm' and looks as follows (the fields B,D,H,J,K,L,O are selected):<\/p>\n

\"Performance
\n
\nThe following parameters will be interesting for us: <\/p>\n

Mem overcommit avg<\/b> \u2014 the average memory overcommitment on the host over 1, 5, and 15 minutes. If above zero, it is a reason to check what is happening, but it does not always indicate problems.<\/p>\n

In the lines PMEM\/MB<\/b> and VMKMEM\/MB<\/b> \u2014 information about the physical memory of the server and the memory available to the VMkernel. Of interest here is the minfree value (in MB), the memory state of the host (in our case, high).<\/p>\n

In the line NUMA\/MB<\/b> you can see the distribution of memory across NUMA nodes (sockets). In this example, the distribution is uneven, which is generally not very good.<\/p>\n

Next is the general server statistics on memory reclamation techniques:<\/p>\n

PSHARE\/MB<\/b> \u2014 this is the TPS statistics;<\/p>\n

SWAP\/MB<\/b> \u2014 statistics on Swap usage;<\/p>\n

ZIP\/MB<\/b> \u2014 statistics on memory page compression;<\/p>\n

MEMCTL\/MB<\/b> \u2014 statistics on Balloon Driver usage.<\/p>\n

For individual VMs, we may be interested in the following information. I have hidden the VM names to avoid confusing the audience :). If the ESXTOP metric is similar to the counter in vSphere, I provide the corresponding counter. <\/p>\n

MEMSZ<\/b> \u2014 amount of memory configured on the VM (MB).
\nMEMSZ = GRANT + MCTLSZ + SWCUR + untouched.<\/p>\n

GRANT<\/b> \u2014 Granted in MB.<\/p>\n

TCHD<\/b> \u2014 Active in MB.<\/p>\n

MCTL?<\/b> \u2014 whether the Balloon Driver is set on the VM.<\/p>\n

MCTLSZ<\/b> \u2014 Balloon in MB.<\/p>\n

MCTLGT<\/b> \u2014 amount of RAM (MB) that ESXi wants to reclaim from the VM through the Balloon Driver (Memctl Target).<\/p>\n

MCTLMAX<\/b> \u2014 maximum amount of RAM (MB) that ESXi can reclaim from the VM through the Balloon Driver.<\/p>\n

SWCUR<\/b> \u2014 current amount of RAM (MB) given to the VM from the Swap file. <\/p>\n

SWGT<\/b> \u2014 amount of RAM (MB) that ESXi wants to give to the VM from the Swap file (Swap Target).<\/p>\n

You can also view more detailed information about the NUMA topology of the VMs through ESXTOP. To do this, select the fields D,G:<\/p>\n

\"Performance
\n
\nNHN<\/b> \u2013 NUMA nodes on which the VM is located. Here you can immediately notice wide VMs that do not fit on a single NUMA node.<\/p>\n

NRMEM<\/b> \u2013 how many megabytes of memory the VM is taking from a remote NUMA node.<\/p>\n

NLMEM<\/b> \u2013 how many megabytes of memory the VM is taking from the local NUMA node.<\/p>\n

N%L<\/b> \u2013 percentage of VM memory on the local NUMA node (if less than 80% \u2014 performance issues may arise).<\/p>\n

Memory on the hypervisor<\/h3>\n

\nIf CPU counters on the hypervisor usually do not represent particular interest, the situation is opposite with memory. High Memory Usage on a VM does not always indicate a performance problem, but high Memory Usage on the hypervisor triggers memory management techniques and causes performance issues for the VM. It's important to monitor Host Memory Usage alarms and prevent VMs from going into Swap.<\/p>\n

\"Performance
\n
\n\"Performance
\n<\/p>\n

Unswap<\/h3>\n

\nIf a VM has entered Swap, its performance drops significantly. The traces of Ballooning and compression quickly disappear after free RAM becomes available on the host, but the virtual machine is in no hurry to return from Swap to the host's RAM.
\nBefore ESXi 6.0, the only reliable and quick way to get VMs out of Swap was to reboot (more precisely, to turn off\/on the container). Starting with ESXi 6.0, there emerged an unofficial yet effective method to remove VMs from Swap. During one conference, I had the chance to talk to one of the VMware engineers responsible for the CPU Scheduler. He confirmed that this method is indeed effective and safe. In our experience, we have not encountered any issues with it.<\/p>\n

Commands for getting VMs out of Swap described<\/a><\/noindex> Duncan Epping. I won't repeat the detailed description; I will just provide an example of its usage. As seen in the screenshot, after a while of executing the specified command, the Swap on the VM disappears.<\/p>\n

\"Performance
\n<\/p>\n

Tips for managing memory on ESXi<\/h3>\n

\nFinally, here are a few tips that can help you avoid VM performance issues due to memory:<\/p>\n