This article aims to discuss the features of All Flash arrays from AccelStor with one of the most popular virtualization platforms – VMware vSphere. In particular, we will highlight the parameters that can help maximize the benefits of using such a powerful tool as All Flash.
AccelStor NeoSapphire™ All Flash arrays are or node devices based on SSDs with a fundamentally different approach to the implementation of data storage concepts and access organization using proprietary technology instead of the widely used RAID algorithms. The arrays provide block access for hosts through Fibre Channel or iSCSI interfaces. To be fair, models with iSCSI interfaces also have file access as a nice bonus. However, in this article, we will focus on the application of block protocols, which are the most efficient for All Flash.
The entire process of deploying and subsequently configuring the collaboration between the AccelStor array and the VMware vSphere virtualization system can be divided into several stages:
- Implementing the connection topology and configuring the SAN network;
- Configuring the All Flash array;
- Configuring the ESXi hosts;
- Configuring the virtual machines.
The equipment used for examples included AccelStor NeoSapphire™ arrays with Fibre Channel and iSCSI interfaces. The base software was VMware vSphere 6.7U1.
Before deploying the systems described in this article, it is highly recommended to review VMware's documentation regarding performance issues ( ) and iSCSI settings ()
Connection topology and SAN network configuration
The main components of the SAN network are HBA adapters in the ESXi hosts, SAN switches, and array nodes. A typical topology of such a network would look like this:
The term Switch here refers to either a separate physical switch or a set of switches (Fabric), as well as a device shared between different services (VSAN in the case of Fibre Channel and VLAN in the case of iSCSI). Using two independent switches/Fabric will eliminate a potential point of failure.
Directly connecting hosts to an array is supported but strongly discouraged. The performance of All Flash arrays is quite high, and to achieve maximum speed, all array ports need to be utilized. Therefore, having at least one switch between the hosts and NeoSapphire™ is essential.
Having two ports on the host HBA is also a mandatory requirement to achieve maximum performance and ensure fault tolerance.
When using a Fibre Channel interface, zoning configuration is required to prevent possible collisions between initiators and targets. Zones are built on the principle of "one initiator port – one or more array ports."
If iSCSI is used with a shared switch, it is essential to isolate iSCSI traffic within a separate VLAN. It is also strongly recommended to enable Jumbo Frames (MTU = 9000) to increase packet sizes and, thereby, reduce the overhead in transmission. However, it is important to remember that for proper operation, the MTU setting must be changed on all network components along the "initiator-switch-target" chain.
Configuring the All Flash array
The array is delivered to clients with groups already formed. . Therefore, no actions are required to combine drives into a single structure. It is sufficient to create volumes of the required size and in the necessary quantities.
For convenience, there is functionality for batch creation of several volumes of a specified size at once. By default, 'thin' volumes are created, as this allows for more efficient use of available storage space (including support for Space Reclamation). From a performance perspective, the difference between 'thin' and 'thick' volumes does not exceed 1%. However, if you need to "squeeze every last drop" from the array, any 'thin' volume can always be converted to 'thick'. But it should be noted that this operation is irreversible.
Next, it is necessary to "publish" the created volumes and assign access rights to them from the hosts using ACL (IP addresses for iSCSI and WWPN for FC) and physical separation by array ports. For iSCSI models, this is done by creating a Target.
For FC models, publication occurs by creating a LUN for each array port.
To speed up the configuration process, hosts can be grouped together. Moreover, if a multi-port FC HBA is used on the host (which is most commonly the case in practice), the system automatically recognizes that the ports of such an HBA belong to a single host due to WWPNs differing by one. Batch creation of Target/LUN is also supported for both interfaces.
An important note when using the iSCSI interface is to create several targets for the volumes immediately to increase performance, as the queue on the target cannot be altered, and it will essentially become a bottleneck.
Configuring ESXi Hosts
From the perspective of ESXi hosts, the basic configuration is carried out according to a quite expected scenario. The sequence of actions for iSCSI connection:
- Add Software iSCSI Adapter (not required if it is already added, or if a Hardware iSCSI Adapter is used);
- Create a vSwitch through which iSCSI traffic will pass, and add physical uplinks and VMkernel to it;
- Add the array addresses to Dynamic Discovery;
- Create a Datastore
Some important points:
- In general, it is certainly possible to use an existing vSwitch, but in the case of a separate vSwitch, managing the host settings will be significantly easier.
- It is essential to separate Management traffic and iSCSI over different physical links and/or VLANs to avoid performance issues.
- The IP addresses of the VMkernel and the corresponding ports of the All Flash array must be within the same subnet again due to performance considerations.
- To ensure fault tolerance, VMware rules state that the vSwitch must have at least two physical uplinks
- If Jumbo Frames are used, it is necessary to change the MTU both on the vSwitch and on the VMkernel
- It will not be superfluous to remind that according to VMware's recommendations for physical adapters that will be used for iSCSI traffic, it is necessary to configure Teaming and Failover. In particular, each VMkernel should operate through only one uplink, and the second uplink must be set to unused mode. For redundancy, it is necessary to add two VMkernels, each of which will operate through its own uplink.
VMkernel Adapter (vmk#)
Physical Network Adapter (vmnic#)
vmk1 (Storage01)
Active Adapters
vmnic2
Unused Adapters
vmnic3
vmk2 (Storage02)
Active Adapters
vmnic3
Unused Adapters
vmnic2
No preliminary actions are required to connect via Fibre Channel. You can immediately create a Datastore.
After creating the Datastore, ensure that the Round Robin policy is used for paths to the Target/LUN as it is the most performant.
By default, VMware's settings provide for this policy to operate as follows: 1000 requests through the first path, the next 1000 requests through the second path, and so on. This interaction between the host and a dual-controller array will be imbalanced. Therefore, we recommend setting the parameter Round Robin policy = 1 through Esxcli/PowerCLI.
Parameters
For Esxcli:
- List available LUNs
esxcli storage nmp device list
- Copy Device Name
- Change Round Robin Policy
esxcli storage nmp psp roundrobin deviceconfig set —type=iops —iops=1 —device="Device_ID"
Most modern applications are designed for exchanging large data packets to maximize bandwidth utilization and reduce load on the central processor. Hence, ESXi by default sends I/O requests to the storage device in chunks of up to 32767KB. However, for some scenarios, exchanging smaller packets may be more efficient. For AccelStor arrays, these scenarios are:
- The virtual machine uses UEFI instead of Legacy BIOS
- vSphere Replication is used
For such scenarios, it is recommended to change the parameter Disk.DiskMaxIOSize to 4096.
For iSCSI connections, it is recommended to change the Login Timeout parameter to 30 (default is 5) to increase connection stability and disable the DelayedAck acknowledgment delay for forwarded packets. Both options are located in vSphere Client: Host → Configure → Storage → Storage Adapters → Advanced Options for the iSCSI adapter.
A crucial aspect is the number of volumes used for the datastore. It is clear that for ease of management, there may be a desire to create one large volume for the entire array. However, having multiple volumes and, consequently, datastores positively affects overall performance (more on queues further in the text). Therefore, we recommend creating at least two volumes.
Not long ago, VMware advised limiting the number of virtual machines on a single datastore again, with the aim of achieving the highest possible performance. However, this issue is not as pressing now, especially with the rise of VDI. But this does not negate the long-standing rule — distribute virtual machines that require intensive I/O across different datastores. To determine the optimal number of VMs on one volume, nothing beats conducting within your own infrastructure.
Configuring Virtual Machines
There are no specific requirements for configuring virtual machines; rather, they are quite standard:
- Use the highest possible VM version (compatibility)
- Be careful when setting the RAM size during dense placement of virtual machines, for example, in VDI (as by default, a swap file of comparable size to the RAM is created at startup, consuming usable capacity and affecting overall performance)
- Use the most performant I/O versions of adapters: network type VMXNET 3 and SCSI type PVSCSI
- Use the Thick Provision Eager Zeroed disk type for maximum performance and Thin Provisioning for the best space utilization
- If possible, limit the operation of non-critical I/O machines using the Virtual Disk Limit
- Always install VMware Tools
Remarks on queues
The queue (or Outstanding I/Os) is the number of I/O requests (SCSI commands) waiting to be processed at any given time by a specific device/application. In the case of queue overflow, QFULL errors are triggered, which ultimately manifests as an increase in latency. When using disk (spindle) storage systems, in theory, the higher the queue, the better their performance. However, one should not overdo it, as this can easily lead to QFULL errors. For All Flash systems, on the one hand, things are simpler: the array typically has much lower latencies and therefore often does not require adjusting queue sizes separately. On the other hand, in certain usage scenarios (strong imbalances in I/O demands for specific virtual machines, maximum performance tests, etc.), it is necessary to either adjust queue parameters or at least understand what metrics can be achieved and, most importantly, by what means.
On the AccelStor All Flash array itself, there are no limits regarding volumes or I/O ports. If necessary, even a single volume can utilize all the array's resources. The only limitation on queues exists for iSCSI targets. For this reason, the creation of several (ideally up to 8) targets for each volume was recommended to overcome this limitation. It is also worth repeating that AccelStor arrays are highly performant solutions. Therefore, all interface ports of the system should be utilized to achieve maximum speed.
From the perspective of the ESXi host, the situation is completely different. The host itself employs a practice of equal access to resources for all participants. Therefore, there are separate I/O queues for the guest OS and HBA. The queues for the guest OS are combined from the queues to the virtual SCSI adapter and virtual disk:
The queue to the HBA depends on the specific type/vendor:
The overall performance of the virtual machine will be determined by the lowest value of the queue depth limit among the host components.
These values allow us to assess the performance metrics that we can achieve in different configurations. For example, we want to know the theoretical performance of a virtual machine (independent of the block) with a latency of 0.5ms. Then its IOPS = (1,000 / latency) * Outstanding I/Os (Queue Depth limit)
Examples
Example 1
- FC Emulex HBA Adapter
- One VM on datastore
- VMware Paravirtual SCSI Adapter
Here, the Queue Depth limit is defined by the Emulex HBA. Therefore, IOPS = (1000 / 0.5) * 32 = 64K
Example 2
- VMware iSCSI Software Adapter
- One VM on datastore
- VMware Paravirtual SCSI Adapter
Here, the Queue Depth limit is defined by the Paravirtual SCSI Adapter. Therefore, IOPS = (1000 / 0.5) * 64 = 128K
The top models of All Flash arrays from AccelStor (for example, ) can deliver a write performance of 700K IOPS with a 4K block. At this block size, it is quite clear that a single virtual machine cannot load such an array. For this, it will take 11 (for example 1) or 6 (for example 2) virtual machines.
As a result, with the correct configuration of all described components of the virtual data center, one can achieve quite impressive performance results.
4K Random, 70% Read / 30% Write
In reality, the real world is much more complex to describe with a simple formula. On one host, there are always many virtual machines with different configurations and I/O requirements. Moreover, the host processor handles I/O, and its power is not infinite. Thus, to unlock the full potential of the same in reality, three hosts will be needed. Plus, applications running inside virtual machines make their own adjustments. Therefore, for accurate sizing, we suggest All Flash arrays within the client's infrastructure on real current tasks.
Source: habr.com
