Hello,
vSAN Configuration :
4 Nodes HY.
3 DiskGroups per Node (1 SSD / 4 HDD)
When we migrate VM (Storage vMotion) from traditionnal Storage (iSCSI Strorage Array) to vSAN, some VMs (already on vSAN Datastore) are severely impacted.
During the svmotion, we can see some congestions (apparently it's normal during migration, cf : VMware Knowledge Base).
But the issue is Write latency (above 100ms) during the migration on some VMs (not all).
Write latency at vDisk level on VM :
Do you know if it's normal to have so many write latency during Migration with HY Configuration?
Is there a way to optimise the migration (without VM downtime) so that VMs already on vSAN Datastore are not affected?
Thank you !
Hello lolo31,
"To be honest, i just want to know if it's common to get latency on some VM when migrating VM to a vSAN Datastore."
Short answer - Yes. If you push any storage to its limitations contention for resources may result in latencies and performance impact on VMs also using this storage - SvMotion are straight write-operations and this is not Hybrid-clusters forté.
Have you tried migrating VMs with less priority button selected?
Are you moving multiple VMs at once?
Have you tried using advanced SvMotion to just migrate single disks of VMs one at a time instead of all the VMs disks?
What exact build of ESXi/vSAN is in use here? Check from host summary in web client or via CLI:
# vmware - vl
H730P controllers are fairly decent but in the past they have endured a number of driver and firmware issues which definitely could impact performance, including Power-on Resets, H:0x7 resets and H:0x8 aborts - if any of these are occurring it should be relatively easy to see and count in the vmkernel.log:
# grep H: /var/log/vmkernel.log | awk -F failed '{print $2}' | sort | uniq -c
http://www.virten.net/vmware/esxi-scsi-sense-code-decoder
Note that H730P have specific recommendations and configuration requirements in order to be expected to function normally:
No VMFS devices on the same controller as vSAN drives (even for boot or logging!):
Certain advanced configuration settings may need to be set (configured automatically from ESXi 6.0 U3):
HCL listing for these controllers (SSID:1f48 device has the same drivers/firmware):
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vsanio&productid=34857
You can determine the firmware for these controllers from iDRAC and likely also from inspecting the boot log:
# zcat /var/log/boot.gz | grep 'lsi_mr3'
And driver:
# vmkload_mod -s lsi_mr3 | grep Version
HDDs are using correct Firmware (TT31):
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=hdd&productid=39279
As are the SSDs (Drive Firmware is MINIMUM version and DL2B in use is newer than DL29)
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=39786
Hope this helps.
Bob
Curious to know:
Are you using SATA drives?
Are you using 1gig connections instead of 10gig?
Hello,
You can find below the information requested :
- vSAN 6.2
- Server : Dell R730xd
- Raid Controler : PERC H730P Mini
- Disk Group / Capacity and Cache Tier :
Display Name Vendor Model Size (GB) Revision SSD? Offline? Device Max Queue Depth RAID Level
------------ ------ ----- --------- -------- ----- -------- ---------------------- ----------
SEAGATE Serial Attached SCSI Disk (naa.x) SEAGATE ST1200MM0088 1118 TT31 false false 64 NA
SEAGATE Serial Attached SCSI Disk (naa.x) SEAGATE ST1200MM0088 1118 TT31 false false 64 NA
SEAGATE Serial Attached SCSI Disk (naa.x) SEAGATE ST1200MM0088 1118 TT31 false false 64 NA
Local ATA Disk (naa.x) ATA INTEL SSDSC2BX40 373 DL2B true false 64 NA
- 10Gb/s pNIC for vSAN Traffic (i don't remember if there is a dedicated 10 Gb/s port for vSAN Traffic, i have to check) :
Broadcom 57810 DP 10Gb DA/SFP+ CNA
I don't want to check on this forum for HCL Driver and Firmware Version for all this kind of hardware.
To be honest, i just want to know if it's common to get latency on some VM when migrating VM to a vSAN Datastore.
If it's not, i will double check the vSAN configuration, maybe with GSS.
Thanks !
Hello lolo31,
"To be honest, i just want to know if it's common to get latency on some VM when migrating VM to a vSAN Datastore."
Short answer - Yes. If you push any storage to its limitations contention for resources may result in latencies and performance impact on VMs also using this storage - SvMotion are straight write-operations and this is not Hybrid-clusters forté.
Have you tried migrating VMs with less priority button selected?
Are you moving multiple VMs at once?
Have you tried using advanced SvMotion to just migrate single disks of VMs one at a time instead of all the VMs disks?
What exact build of ESXi/vSAN is in use here? Check from host summary in web client or via CLI:
# vmware - vl
H730P controllers are fairly decent but in the past they have endured a number of driver and firmware issues which definitely could impact performance, including Power-on Resets, H:0x7 resets and H:0x8 aborts - if any of these are occurring it should be relatively easy to see and count in the vmkernel.log:
# grep H: /var/log/vmkernel.log | awk -F failed '{print $2}' | sort | uniq -c
http://www.virten.net/vmware/esxi-scsi-sense-code-decoder
Note that H730P have specific recommendations and configuration requirements in order to be expected to function normally:
No VMFS devices on the same controller as vSAN drives (even for boot or logging!):
Certain advanced configuration settings may need to be set (configured automatically from ESXi 6.0 U3):
HCL listing for these controllers (SSID:1f48 device has the same drivers/firmware):
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=vsanio&productid=34857
You can determine the firmware for these controllers from iDRAC and likely also from inspecting the boot log:
# zcat /var/log/boot.gz | grep 'lsi_mr3'
And driver:
# vmkload_mod -s lsi_mr3 | grep Version
HDDs are using correct Firmware (TT31):
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=hdd&productid=39279
As are the SSDs (Drive Firmware is MINIMUM version and DL2B in use is newer than DL29)
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=39786
Hope this helps.
Bob
Hello Bob,
I appreciate the time you took for this complete answer!
I will try to get all informations you request from my customer.
No we didn't try vMotion with Standard Priority, we will try
Yes, only one VM Migration at a time (that was the case on screenshots in my first post).
Regards.
Well, the migration is over and my customer didn't give me the informations i asked...
So, i will highlight your reply for the correct Solution Bob.
Thanks again.