VMware Cloud Community
lolo31
Enthusiast
Enthusiast
Jump to solution

Write Latency on vSAN Datastore when migrating VM to vSAN.

Hello,

vSAN Configuration :

4 Nodes HY.

3 DiskGroups per Node (1 SSD / 4 HDD)

When we migrate VM (Storage vMotion) from traditionnal Storage (iSCSI Strorage Array) to vSAN, some VMs (already on vSAN Datastore) are severely impacted.

During the svmotion, we can see some congestions (apparently it's normal during migration, cf : VMware Knowledge Base​).

But the issue is Write latency (above 100ms) during the migration on some VMs (not all).

cont 00.png

Write latency at vDisk level on VM :

vm lvl 00.png

Do you know if it's normal to have so many write latency during Migration with HY Configuration?

Is there a way to optimise the migration (without VM downtime) so that VMs already on vSAN Datastore are not affected?

Thank you !

Reply
0 Kudos
1 Solution

Accepted Solutions
TheBobkin
Champion
Champion
Jump to solution

Hello lolo31​,

"To be honest, i just want to know if it's common to get latency on some VM when migrating VM to a vSAN Datastore."

Short answer - Yes. If you push any storage to its limitations contention for resources may result in latencies and performance impact on VMs also using this storage - SvMotion are straight write-operations and this is not Hybrid-clusters forté.

Have you tried migrating VMs with less priority button selected?

Are you moving multiple VMs at once?

Have you tried using advanced SvMotion to just migrate single disks of VMs one at a time instead of all the VMs disks?

What exact build of ESXi/vSAN is in use here? Check from host summary in web client or via CLI:

# vmware - vl

H730P controllers are fairly decent but in the past they have endured a number of driver and firmware issues which definitely could impact performance, including Power-on Resets, H:0x7 resets and H:0x8 aborts - if any of these are occurring it should be relatively easy to see and count in the vmkernel.log:

# grep H: /var/log/vmkernel.log | awk -F failed '{print $2}' | sort | uniq -c

http://www.virten.net/vmware/esxi-scsi-sense-code-decoder

Note that H730P have specific recommendations and configuration requirements in order to be expected to function normally:

No VMFS devices on the same controller as vSAN drives (even for boot or logging!):

VMware Knowledge Base

Certain advanced configuration settings may need to be set (configured automatically from ESXi 6.0 U3):

VMware Knowledge Base

HCL listing for these controllers (SSID:1f48 device has the same drivers/firmware):

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vsanio&productid=34857

You can determine the firmware for these controllers from iDRAC and likely also from inspecting the boot log:

# zcat /var/log/boot.gz | grep 'lsi_mr3'

And driver:

# vmkload_mod -s  lsi_mr3 | grep Version

HDDs are using correct Firmware (TT31):

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=hdd&productid=39279

As are the SSDs (Drive Firmware is MINIMUM version and DL2B in use is newer than DL29)

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=39786

Hope this helps.

Bob

View solution in original post

6 Replies
Jasemccarty
Immortal
Immortal
Jump to solution

Curious to know:

  • What build of vSAN you are using
  • What your the cache device size is in your disk groups
  • What your networking configuration looks like (VDS, 10Gb, etc)
Jase McCarty - @jasemccarty
Reply
0 Kudos
srodenburg
Expert
Expert
Jump to solution

Are you using SATA drives?

Are you using 1gig connections instead of 10gig?

Reply
0 Kudos
lolo31
Enthusiast
Enthusiast
Jump to solution

Hello,

You can find below the information requested :

- vSAN 6.2

- Server : Dell R730xd

- Raid Controler : PERC H730P Mini

- Disk Group / Capacity and Cache Tier :

Display Name                                              Vendor    Model             Size (GB)  Revision  SSD?   Offline?  Device Max Queue Depth  RAID Level

                ------------                                              ------    -----             ---------  --------  -----  --------  ----------------------  ----------

                SEAGATE Serial Attached SCSI Disk (naa.x)  SEAGATE   ST1200MM0088      1118       TT31      false  false     64                      NA

                SEAGATE Serial Attached SCSI Disk (naa.x)  SEAGATE   ST1200MM0088      1118       TT31      false  false     64                      NA

                SEAGATE Serial Attached SCSI Disk (naa.x)  SEAGATE   ST1200MM0088      1118       TT31      false  false     64                      NA

                Local ATA Disk (naa.x)                     ATA       INTEL SSDSC2BX40  373        DL2B      true   false     64                      NA           

- 10Gb/s pNIC for vSAN Traffic (i don't remember if there is a dedicated 10 Gb/s port for vSAN Traffic, i have to check) :

Broadcom 57810 DP 10Gb DA/SFP+ CNA

I don't want to check on this forum for HCL Driver and Firmware Version for all this kind of hardware.

To be honest, i just want to know if it's common to get latency on some VM when migrating VM to a vSAN Datastore.

If it's not, i will double check the vSAN configuration, maybe with GSS.

Thanks !

Reply
0 Kudos
TheBobkin
Champion
Champion
Jump to solution

Hello lolo31​,

"To be honest, i just want to know if it's common to get latency on some VM when migrating VM to a vSAN Datastore."

Short answer - Yes. If you push any storage to its limitations contention for resources may result in latencies and performance impact on VMs also using this storage - SvMotion are straight write-operations and this is not Hybrid-clusters forté.

Have you tried migrating VMs with less priority button selected?

Are you moving multiple VMs at once?

Have you tried using advanced SvMotion to just migrate single disks of VMs one at a time instead of all the VMs disks?

What exact build of ESXi/vSAN is in use here? Check from host summary in web client or via CLI:

# vmware - vl

H730P controllers are fairly decent but in the past they have endured a number of driver and firmware issues which definitely could impact performance, including Power-on Resets, H:0x7 resets and H:0x8 aborts - if any of these are occurring it should be relatively easy to see and count in the vmkernel.log:

# grep H: /var/log/vmkernel.log | awk -F failed '{print $2}' | sort | uniq -c

http://www.virten.net/vmware/esxi-scsi-sense-code-decoder

Note that H730P have specific recommendations and configuration requirements in order to be expected to function normally:

No VMFS devices on the same controller as vSAN drives (even for boot or logging!):

VMware Knowledge Base

Certain advanced configuration settings may need to be set (configured automatically from ESXi 6.0 U3):

VMware Knowledge Base

HCL listing for these controllers (SSID:1f48 device has the same drivers/firmware):

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vsanio&productid=34857

You can determine the firmware for these controllers from iDRAC and likely also from inspecting the boot log:

# zcat /var/log/boot.gz | grep 'lsi_mr3'

And driver:

# vmkload_mod -s  lsi_mr3 | grep Version

HDDs are using correct Firmware (TT31):

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=hdd&productid=39279

As are the SSDs (Drive Firmware is MINIMUM version and DL2B in use is newer than DL29)

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=39786

Hope this helps.

Bob

lolo31
Enthusiast
Enthusiast
Jump to solution

Hello Bob,

I appreciate the time you took for this complete answer!

I will try to get all informations you request from my customer.

No we didn't try vMotion with Standard Priority, we will try Smiley Happy

Yes, only one VM Migration at a time (that was the case on screenshots in my first post).

Regards.

Reply
0 Kudos
lolo31
Enthusiast
Enthusiast
Jump to solution

Well, the migration is over and my customer didn't give me the informations i asked...

So, i will highlight your reply for the correct Solution Bob.

Thanks again.

Reply
0 Kudos