How to adjust the Round Robin IOPS limit from default 1000 to 1 ESXi 7

In this blog we will learn how to change the Round Robin Path Selection Policy (VMW_PSP_RR) on FC and iSCSI LUNs in a VMware environment to better balance the I/O load across all active storage paths.

Modifying the IOPS limitation to 1 will improve performance where an active storage path might have several queued I/Os. In addition, setting the limitation to 1 allows other active storage paths to service I/O requests.  This will have the benefit of reduced latency and increased throughput.

ESXi Round Robin PSP supports two types of limits:

  • IOPS limit: The Round Robin PSP defaults to an IOPS limit with a value of 1000. In this default case, a new path is used after 1000 I/O operations are issued.
  • Bytes limit: The bytes limit is an alternative to the IOPS limit. The bytes limit allows for a specified amount of bytes to be transferred before the path is switched. (Default = 10485760 bytes)
  • Essentially, Round Robin will attempt to re-balance after every 1000 I/Os or 10485760 bytes.

Reference:

https://kb.vmware.com/s/article/2069356

Note: You do not need to restart the host for the changes to take effect.

Configure ESXi Host

I now have to enable the SSH service then SSH into your host and run the following command to list all of the storage SCSI devices. This will include all FC, iSCSI and local block storage mounted on the host.

esxcli storage nmp device list | grep naa
[root@R720-HOST-01:~] esxcli storage nmp device list | grep naa
naa.6589cfc0000006809342a12039572e04
   Device Display Name: TrueNAS Fibre Channel Disk (naa.6589cfc0000006809342a12039572e04)
naa.6589cfc000000816db54001e3f0d2264
   Device Display Name: TrueNAS Fibre Channel Disk (naa.6589cfc000000816db54001e3f0d2264)

Get info about a LUN by running the below command on one of you LUN IDs

esxcli storage nmp device list -d naa.6589cfc0000006809342a12039572e04
[root@R720-HOST-01:~] esxcli storage nmp device list -d naa.6589cfc0000006809342a12039572e04
naa.6589cfc0000006809342a12039572e04
   Device Display Name: TrueNAS Fibre Channel Disk (naa.6589cfc0000006809342a12039572e04)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=iops,iops=1000,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config: policy=iops;iops=1000;bytes=10485760;samplingCycles=16;latencyEvalTime=180000;useANO=0;
   Working Paths: vmhba3:C0:T1:L1, vmhba2:C0:T0:L1
   Is USB: false

As you can see the IOPS are set to 1000. We want to change IOPS to 1 for all of the FC LUNs. This can be done with one command. As you can see the LUN IDs for TureNAS all start with same string of numbers. We can pipe into grep to select only LUNs that have IDs that begin with naa.6589 and set IOPS to 1.

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep naa.6589`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done

I can now check that the IOPS policy has been changed to 1.

esxcli storage nmp device list -d naa.6589cfc0000006809342a12039572e04
[root@R720-HOST-01:~] esxcli storage nmp device list -d naa.6589cfc0000006809342a12039572e04
naa.6589cfc0000006809342a12039572e04
   Device Display Name: TrueNAS Fibre Channel Disk (naa.6589cfc0000006809342a12039572e04)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0; lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config: policy=iops;iops=1;bytes=10485760;samplingCycles=16;latencyEvalTime=180000;useANO=0;
   Working Paths: vmhba3:C0:T1:L1, vmhba2:C0:T0:L1
   Is USB: false

There you go, it could not be any easier to achieve better storage performance.

You can use the below command to find out the features supported by the storage server.

esxcli storage core device list -d naa.6589cfc0000006809342a12039572e04
[root@R720-HOST-01:~]  esxcli storage core device list -d naa.6589cfc0000006809342a12039572e04
naa.6589cfc0000006809342a12039572e04
   Display Name: TrueNAS Fibre Channel Disk (naa.6589cfc0000006809342a12039572e04)
   Has Settable Display Name: true
   Size: 2915041
   Device Type: Direct-Access 
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.6589cfc0000006809342a12039572e04
   Vendor: TrueNAS 
   Model: iSCSI Disk      
   Revision: 0123
   SCSI Level: 7
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: true
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters: 
   VAAI Status: supported
   Other UIDs: vml.010001000039306231316333653366323630303100695343534920
   Is Shared Clusterwide: true
   Is SAS: false
   Is USB: false
   Is Boot Device: false
   Device Max Queue Depth: 61
   No of outstanding IOs with competing worlds: 32
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

Leave a Comment

Your email address will not be published. Required fields are marked *