This article discusses performance counters for RAM in vSphere.
Memory issues seem to be more straightforward than CPU ones: if a VM encounters performance problems, they are hard to miss. However, once they appear, it's much more challenging to address them. But let's take it step by step.
A Bit of Theory
The RAM for virtual machines is drawn from the memory of the servers on which the VMs operate. This is quite obvious :) If the server's RAM is insufficient for all requests, ESXi begins to apply memory reclamation techniques. Otherwise, the operating systems of the VMs would crash with access errors to RAM.
The techniques ESXi applies depend on the memory load:
Memory State
Threshold
Actions
High
400% of minFree
Once the upper threshold is reached, large memory pages are broken down into smaller ones (TPS operates in standard mode).
Clear
100% of minFree
Large memory pages are broken down into smaller ones, TPS operates forcefully.
, which is located within the driver’s codebase itself. It uses its own
64% of minFree
TPS + Balloon
Hard
32% of minFree
TPS + Compress + Swap
Low
16% of minFree
Compress + Swap + Block
minFree is the amount of RAM necessary for the hypervisor to operate.
Until ESXi 4.1, minFree was by default fixed at 6% of the server's RAM (the percentage could be changed using the Mem.MinFreePct option in ESXi). In later versions, due to the increasing amounts of memory on servers, minFree began to be calculated based on the host's memory size rather than as a fixed percentage.
The default minFree value is calculated as follows:
Percentage of memory reserved for minFree
Memory range
6%
0-4 GB
4%
4-12 GB
2%
12-28 GB
1%
Remaining memory
For example, for a server with 128 GB of RAM, the MinFree value would be as follows:
MinFree = 245.76 + 327.68 + 327.68 + 1024 = 1925.12 MB = 1.88 GB
The actual value may differ by a few hundred MB, depending on the server and memory.
Percentage of memory reserved for minFree
Memory range
Value for 128 GB
6%
0-4 GB
245.76 MB
4%
4-12 GB
327.68 MB
2%
12-28 GB
327.68 MB
1%
Remaining memory (100 GB)
1024 MB
Typically, the High state is considered normal for production environments. For testing and development environments, Clear/Soft states may be acceptable. If less than 64% of MinFree memory remains on the host, VMs running on it will undoubtedly experience performance issues.
Specific memory reclamation techniques are applied in each state, starting with TPS, which has a minimal impact on VM performance, up to swapping. I'll explain them in more detail.
Transparent Page Sharing (TPS). TPS is, roughly speaking, the deduplication of virtual machine memory pages on the server.
ESXi looks for identical virtual machine memory pages by calculating and comparing the hash values of the pages and removes duplicates, replacing them with references to the same page in the server's physical memory. As a result, physical memory consumption decreases, allowing for some memory oversubscription without significantly impacting performance.
This mechanism only works for memory pages sized at 4 KB (small pages). The hypervisor doesn't even attempt to deduplicate 2 MB (large pages) as the chances of finding identical pages of that size are low.
By default, ESXi allocates memory for large pages. Fragmentation of large pages into small ones begins when the high state threshold is reached and occurs forcefully when the clear state is achieved (see the hypervisor state table).
If you want TPS to start working without waiting for the host's memory to fill up, you need to set the value in the Advanced Options of ESXi to “Mem.AllocGuestLargePage” to 0 (default is 1). Then, the allocation of large memory pages for virtual machines will be disabled.
Since December 2014, TPS between VMs has been disabled by default in all ESXi releases due to the discovery of a vulnerability that theoretically allows one VM to access the memory of another VM. Details can be found here. I have not encountered practical implementation information about exploiting the TPS vulnerability.
The TPS policy is controlled through the advanced option “Mem.ShareForceSalting” on ESXi:
0 — Inter-VM TPS. TPS works for pages of different VMs;
1 – TPS for VMs with the same value of “sched.mem.pshare.salt” in VMX;
2 (default) – Intra-VM TPS. TPS works for pages within the VM.
It definitely makes sense to disable large pages and enable Inter-VM TPS on test environments. This can also be useful for environments with a large number of similar VMs. For example, in VDI environments, the savings in physical memory can reach tens of percent.
Memory Ballooning. Ballooning is no longer as harmless and transparent for the VM operating system as TPS. However, with proper application, one can live and even work with Ballooning.
Along with VMware Tools, a special driver called the Balloon Driver (also known as vmmemctl) is installed on the VM. When the hypervisor runs low on physical memory and transitions to a Soft state, ESXi asks the VM to return unused RAM through this Balloon Driver. The driver, in turn, operates at the level of the operating system and requests free memory from it. The hypervisor sees which pages of physical memory the Balloon Driver occupies, takes memory from the virtual machine, and returns it to the host. There are no issues with the OS's operation since the memory is occupied by the Balloon Driver at the OS level. By default, the Balloon Driver can reclaim up to 65% of the VM's memory.
If VMware Tools are not installed on the VM or Ballooning is disabled (not recommended, but there are situations), the hypervisor immediately resorts to more severe memory reclamation techniques. :), the hypervisor immediately moves to stricter memory reclamation techniques. Conclusion: ensure VMware Tools are installed on the VM.
The operation of the Balloon Driver can be checked from the OS through VMware Tools..
Memory Compression. This technique is applied when ESXi reaches a Hard state. As the name suggests, ESXi attempts to compress 4 KB memory pages to 2 KB in order to free up a bit of space in the server's physical memory. This technique significantly increases the access time to the contents of the VM's memory pages, as the page needs to be decompressed first. Sometimes not all pages can be compressed, and the process itself takes some time. Therefore, this technique is not very effective in practice.
Memory Swapping. After a brief phase of Memory Compression, ESXi almost inevitably (unless VMs have migrated to other hosts or powered off) transitions to Swapping. If there is very little memory left (Low state), the hypervisor also stops allocating memory pages to the VMs, which may cause problems in the guest operating systems of the VMs.
This is how Swapping works. When a virtual machine is powered on, a file with the .vswp extension is created for it. Its size equals the unreserved RAM of the VM: the difference between the configured and reserved memory. During Swapping, ESXi unloads the memory pages of the virtual machine into this file and begins to use it instead of the physical memory of the server. Naturally, this type of "operational" memory is several orders of magnitude slower than actual memory, even if the .vswp file is on fast storage.
Unlike Ballooning, where unused pages are reclaimed from the VM, during Swapping, pages that are actively used by the OS or applications within the VM may be written to disk. As a result, the VM's performance may drop to the point of stalling. The VM technically continues to operate, and it can at least be properly shut down from the OS. If you are patient 😉
If the VMs have gone into Swap, this is an abnormal situation that should be avoided whenever possible.
Key memory performance counters of the virtual machine
Now we have reached the main point. The following counters are available for monitoring memory states in VMs:
Active — shows the amount of memory (KB) that the VM had access to in the previous measurement period.
Usage — the same as Active, but expressed as a percentage of the VM's configured RAM. Calculated using the following formula: active ÷ virtual machine configured memory size.
High Usage and Active, respectively, are not always indicators of VM performance issues. If the VM is aggressively using memory (at least accessing it), it does not mean that there is not enough memory. Rather, this is a reason to check what is happening in the OS.
There is a standard Alarm for Memory Usage for VMs:
Shared — the amount of memory used by the VM that has been deduplicated using TPS (within the VM or between VMs).
Granted — the amount of physical memory of the host (KB) that has been allocated to the VM. Includes Shared.
Consumed (Granted — Shared) — the amount of physical memory (KB) that the VM consumes from the host. Does not include Shared.
If some memory of the VM is not drawn from the host's physical memory but from the swap file or is reclaimed from the VM via Balloon Driver, this amount is not included in Granted and Consumed.
High values for Granted and Consumed are completely normal. The operating system gradually takes memory from the hypervisor and does not give it back. Over time, for an actively running VM, the values of these counters approach the size of the configured memory and stay there.
Zero — the amount of memory for the VM (KB) that contains zeros. This memory is considered free by the hypervisor and can be given to other virtual machines. Once the guest OS writes anything to the zeroed memory, it transitions to Consumed and does not return.
Reserved Overhead — the amount of memory for the VM (KB) reserved by the hypervisor for the VM's operation. This is a small amount, but it must be available on the host; otherwise, the VM will not start.
Balloon — the amount of memory (KB) taken from the VM using the Balloon Driver.
Compressed — the amount of memory (KB) that was successfully compressed.
Swapped — the amount of memory (KB) that, due to the lack of physical memory on the server, has moved to disk.
Balloon and other memory reclamation technique counters are zero.
This is what the graph with the counters for a normally functioning VM with 150 GB of RAM looks like.
In the graph below, the VM has clear problems. Below the graph, it can be seen that all the described techniques for managing memory have been used for this VM. The Balloon for this VM is significantly larger than Consumed. In fact, the VM is more likely dead than alive.
ESXTOP
As with the CPU, if we want to quickly assess the situation on the host and its dynamics with intervals up to 2 seconds, we should use ESXTOP.
The ESXTOP screen for Memory is invoked by pressing the key 'm' and looks as follows (the fields B,D,H,J,K,L,O are selected):
The following parameters will be interesting for us:
Mem overcommit avg — the average memory overcommitment on the host over 1, 5, and 15 minutes. If above zero, it is a reason to check what is happening, but it does not always indicate problems.
In the lines PMEM/MB and VMKMEM/MB — information about the physical memory of the server and the memory available to the VMkernel. Of interest here is the minfree value (in MB), the memory state of the host (in our case, high).
In the line NUMA/MB you can see the distribution of memory across NUMA nodes (sockets). In this example, the distribution is uneven, which is generally not very good.
Next is the general server statistics on memory reclamation techniques:
PSHARE/MB — this is the TPS statistics;
SWAP/MB — statistics on Swap usage;
ZIP/MB — statistics on memory page compression;
MEMCTL/MB — statistics on Balloon Driver usage.
For individual VMs, we may be interested in the following information. I have hidden the VM names to avoid confusing the audience :). If the ESXTOP metric is similar to the counter in vSphere, I provide the corresponding counter.
MEMSZ — amount of memory configured on the VM (MB).
MEMSZ = GRANT + MCTLSZ + SWCUR + untouched.
GRANT — Granted in MB.
TCHD — Active in MB.
MCTL? — whether the Balloon Driver is set on the VM.
MCTLSZ — Balloon in MB.
MCTLGT — amount of RAM (MB) that ESXi wants to reclaim from the VM through the Balloon Driver (Memctl Target).
MCTLMAX — maximum amount of RAM (MB) that ESXi can reclaim from the VM through the Balloon Driver.
SWCUR — current amount of RAM (MB) given to the VM from the Swap file.
SWGT — amount of RAM (MB) that ESXi wants to give to the VM from the Swap file (Swap Target).
You can also view more detailed information about the NUMA topology of the VMs through ESXTOP. To do this, select the fields D,G:
NHN – NUMA nodes on which the VM is located. Here you can immediately notice wide VMs that do not fit on a single NUMA node.
NRMEM – how many megabytes of memory the VM is taking from a remote NUMA node.
NLMEM – how many megabytes of memory the VM is taking from the local NUMA node.
N%L – percentage of VM memory on the local NUMA node (if less than 80% — performance issues may arise).
Memory on the hypervisor
If CPU counters on the hypervisor usually do not represent particular interest, the situation is opposite with memory. High Memory Usage on a VM does not always indicate a performance problem, but high Memory Usage on the hypervisor triggers memory management techniques and causes performance issues for the VM. It's important to monitor Host Memory Usage alarms and prevent VMs from going into Swap.
Unswap
If a VM has entered Swap, its performance drops significantly. The traces of Ballooning and compression quickly disappear after free RAM becomes available on the host, but the virtual machine is in no hurry to return from Swap to the host's RAM.
Before ESXi 6.0, the only reliable and quick way to get VMs out of Swap was to reboot (more precisely, to turn off/on the container). Starting with ESXi 6.0, there emerged an unofficial yet effective method to remove VMs from Swap. During one conference, I had the chance to talk to one of the VMware engineers responsible for the CPU Scheduler. He confirmed that this method is indeed effective and safe. In our experience, we have not encountered any issues with it.
Commands for getting VMs out of Swap Duncan Epping. I won't repeat the detailed description; I will just provide an example of its usage. As seen in the screenshot, after a while of executing the specified command, the Swap on the VM disappears.
Tips for managing memory on ESXi
Finally, here are a few tips that can help you avoid VM performance issues due to memory:
- Avoid overcommitting memory in production clusters. It is advisable to always maintain about 20-30% free memory in the cluster so that DRS (and the administrator) has room to maneuver, and VMs do not go into Swap during migrations. Also, keep in mind the reserve for fault tolerance. It is unpleasant when a server fails and VMs are rebooted using HA, while some machines also go into Swap.
- In highly consolidated infrastructures, try NOT to create VMs with memory exceeding half of the host's memory. This will again help DRS to distribute virtual machines across cluster servers without issues. This rule, of course, is not universal : ).
- Monitor the Host Memory Usage Alarm.
- Don't forget to install VMware Tools on the VMs and keep Ballooning enabled.
- Consider enabling Inter-VM TPS and disabling Large Pages in VDI environments and test labs.
- If a VM is experiencing performance issues, check whether it is using memory from a remote NUMA node.
- Get VMs out of Swap as quickly as possible! Among other things, if a VM is in Swap, the storage system suffers for obvious reasons.
That's all I have on memory. Below are articles on the topic for those who want to delve deeper into the details. The next article will focus on storage.
Useful links
Source: habr.com
