Today we will analyze the metrics of the disk subsystem in vSphere. The problem with storage is the most common reason for the slow operation of a virtual machine. If in cases with CPU and RAM troubleshooting ends at the hypervisor level, then in case of problems with the disk, you may have to deal with the data transfer network and storage.
I will analyze the topic using the example of block access to storage systems, although the counters are approximately the same with file access.
Some theory
When talking about the performance of the disk subsystem of virtual machines, they usually pay attention to three parameters related to each other:
- the number of input / output operations (Input / Output Operations Per Second, IOPS);
- throughput (Throughput);
- latency of input / output operations (Latency).
Number of IOPS usually important for random workloads: access to disk blocks located in different places. Databases, business applications (ERP, CRM), etc. can serve as an example of such a load.
Throughput important for sequential loads: access to blocks located one behind the other. For example, file servers (but not always) and video surveillance systems can generate such a load.
Throughput is related to the number of I/O operations as follows:
Throughput = IOPS * Block size, where Block size is the block size.
The block size is quite an important characteristic. Modern versions of ESXi skip blocks up to 32 KB. If the block is even larger, it is divided into several. Not all storage systems can work effectively with such large blocks, so there is a DiskMaxIOSize parameter in Advanced Settings ESXi. With it, you can reduce the maximum block size skipped by the hypervisor (more ). I recommend that you consult with the storage manufacturer before changing this parameter, or at least test the changes on a laboratory bench.
A large block size can be detrimental to storage performance. Even if the number of IOPS and throughput are relatively small, high latencies can be observed with a large block size. So pay attention to this option.
Latency is the most interesting performance parameter. I/O latency for a virtual machine is the sum of:
- delays inside the hypervisor (KAVG, Average Kernel MilliSec/Read);
- the delay given by the data network and storage systems (DAVG, Average Driver MilliSec/Command).
The total latency that is visible in the guest OS (GAVG, Average Guest MilliSec/Command) is the sum of KAVG and DAVG.
GAVG and DAVG are measured and KAVG is calculated: GAVG–DAVG.
Let's take a closer look at KAVG. During normal operation, KAVG should approach zero, or at least be much smaller than DAVG. The only case I know of where KAVG is expectedly high is the IOPS limit on the VM disk. In this case, when you try to exceed the limit, KAVG will increase.
The most significant component of KAVG is QAVG - the time in the queue for processing inside the hypervisor. The rest of the KAVG components are negligible.
The queue in the disk adapter driver and queue to moons has a fixed size. For highly loaded environments, this size can be useful to increase. describes how to increase the queues in the adapter driver (at the same time, the queue to the moons will increase). This setting works when only one VM is working with the moon, which is rare. If there are several VMs on the moon, you must also increase the parameter Disk.SchedNumReqOutstanding (instruction ). By increasing the queue, you decrease QAVG and KAVG respectively.
But, again, first read the documentation from the HBA vendor and test the changes on a lab bench.
The size of the queue to the moon can be affected by the inclusion of the SIOC (Storage I / O Control) mechanism. It provides uniform access to the moon from all servers in the cluster by dynamically changing the queue to the moon on the servers. That is, if a VM is running on one of the hosts, which requires a disproportionate amount of performance (noisy neighbor VM), SIOC reduces the length of the queue to the moon on this host (DQLEN). More .
We figured out KAVG, now a little about DAVG. Everything is simple here: DAVG is the delay introduced by the external environment (data network and storage). Any modern and not very storage system has its own performance counters. To analyze problems with DAVG, it makes sense to look at them. If everything is fine on the ESXi and storage side, check the data network.
To avoid performance issues, choose the right Path Selection Policy (PSP) for your storage system. Almost all modern storage systems support PSP Round-Robin (with or without ALUA, Asymmetric Logical Unit Access). This policy allows you to use all available storage paths. In the case of ALUA, only paths to the controller that owns the moon are used. Not all storage systems on ESXi have default rules that set the Round-Robin policy. If there is no rule for your storage system, use a plug-in from the storage manufacturer that will create the corresponding rule on all cluster hosts, or create the rule yourself. Details .
Also, some storage manufacturers recommend changing the number of IOPS per path from the standard value of 1000 to 1. In our practice, this allowed us to “squeeze” more performance out of the storage system and significantly reduce the time it takes to failover in case of failure or updating controllers. Check with the vendor's recommendations, and if there are no contraindications, then try changing this setting. Details .
The main performance counters of the disk subsystem of the virtual machine
The performance counters of the disk subsystem in vCenter are collected in the Datastore, Disk, Virtual Disk sections:
In section Datastore there are metrics for vSphere disk storages (datastores) on which VM disks lie. Here you will find standard counters for:
- IOPS (Average read/write requests per second),
- throughput (Read/Write rate),
- delays (Read/Write/Highest latency).
From the names of the counters, in principle, everything is clear. Once again, I would like to draw your attention to the fact that here the statistics are not for a specific VM (or VM disk), but general for the entire datastore. In my opinion, it is more convenient to look at these statistics in ESXTOP, at least based on the fact that the minimum measurement period there is 2 seconds.
In section Disk there are metrics for block devices that are used by the VM. There are counters for IOPS of the summation type (the number of I / O operations during the measurement period) and several counters related to block access (Commands aborted, Bus resets). This information, in my opinion, is also more convenient to look at in ESXTOP.
Section virtual disk - the most useful in terms of finding performance problems with the VM disk subsystem. Here you can see the performance for each virtual disk. It is this information that is needed to understand if a particular virtual machine has a problem. In addition to the standard counters for the number of I / O operations, read / write volume and delays, this section contains useful counters that show the block size: Read / Write request size.
In the picture below, a graph of the VM disk performance, where you can see the number of IOPS, latency and block size.
Also, performance metrics can be viewed across the entire datastore if SIOC is enabled. Here you will find basic information on average Latency and IOPS. By default, this information can only be viewed in real time.
ESXTOP
ESXTOP has several screens that provide information about the host disk subsystem as a whole, individual virtual machines and their disks.
Let's start with information on virtual machines. The “Disk VM” screen is called up with the “v” key:
NVDISK is the number of VM disks. To view information for each disk, press “e” and enter the GID of the VM you are interested in.
The meaning of other parameters on this screen is clear from their names.
Another useful screen for troubleshooting is the Disk adapter. Called by the “d” key (fields A,B,C,D,E,G are selected in the picture below):
NPTH – the number of paths to the moons that are visible from this adapter. To get information about each path on an adapter, press "e" and enter the name of the adapter:
AQLEN – maximum queue size on the adapter.
Also on this screen are the delay counters that I talked about above: KAVG/cmd, GAVG/cmd, DAVG/cmd, QAVG/cmd.
On the Disk device screen, which is called by the “u” key, information is presented on individual block devices - moons (fields A, B, F, G, I are selected in the picture below). Here you can see the state of the queue for the moons.
DQLEN – queue size for the block device.
ACTV – number of I/O commands in the ESXi kernel.
QUED – number of I/O commands in the queue.
%USD – ACTV / DQLEN × 100%.
LOAD – (ACTV + QUED) / DQLEN.
If %USD is high, consider increasing the queue. The more commands in the queue, the higher QAVG and, accordingly, KAVG.
Also on the Disk device screen, you can see if VAAI (vStorage API for Array Integration) works on the storage system. To do this, select fields A and O.
The VAAI mechanism allows you to transfer part of the work from the hypervisor directly to the storage system, for example, zeroing, copying blocks or locks.
As you can see in the picture above, VAAI works on this storage system: the Zero and ATS primitives are actively used.
ESXi Disk Optimization Tips
- Pay attention to block size.
- Set the optimal queue size on the HBA.
- Don't forget to enable SIOC on datastores.
- Choose your PSP according to the storage manufacturer's recommendations.
- Make sure VAAI is working.
Useful articles on the topic:
Source: habr.com