Latency troublshooting for specific VM

[SCharchouf]

Hello, I have an issue with a specific VM with latency and Disk very high wait times (WA) reported from vmstat

I created the below script and not sure if it's correct or No, could you please assist me on the same?

I would like to retreive all info in CSV output espacially naa.xxxxxxxx

$VMName = "MyVM"

$stat = 'datastore.read.average','datastore.write.average'

$dsTab = @{}

$esx = Get-VM -Name $VMName

Get-Datastore -RelatedObject $esx | where{$_.Type -eq 'VMFS'} |

ForEach-Object -Process {

    $dsTab.Add($_.ExtensionData.Info.Vmfs.Uuid,$_.Name)

}

Get-Stat -Entity $esx -Stat $stat -Realtime -MaxSamples 1 |

Group-Object -Property Instance |

ForEach-Object -Process {

    [pscustomobject]@{

        DS = $dsTab[$_.Name]

        ReadKBps = $_.Group | where{$_.MetricId -eq 'datastore.read.average'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

        WriteKBps = $_.Group | where{$_.MetricId -eq 'datastore.write.average'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

    }

}

[LucD]

I'm not sure what you are trying to do with this script.
You are retrieving the rate of reading and writing (in kBPS) data to datastore..average

If you are investigating a latency issue, shouldn't those metrics rather be datastore.totalReadLatency and datastore.totalWriteLatency.average?

[SCharchouf]

It's my bad, you are right

$VMName = "MyVM"

$stat = 'datastore.totalReadLatency','datastore.totalWriteLatency.average'

$dsTab = @{}

$esx = Get-VM -Name $VMName

Get-Datastore -RelatedObject $esx | where{$_.Type -eq 'VMFS'} |

ForEach-Object -Process {

    $dsTab.Add($_.ExtensionData.Info.Vmfs.Uuid,$_.Name)

}

Get-Stat -Entity $esx -Stat $stat -Realtime -MaxSamples 1 |

Group-Object -Property Instance |

ForEach-Object -Process {

    [pscustomobject]@{

        DS = $dsTab[$_.Name]

        ReadKBps = $_.Group | where{$_.MetricId -eq 'datastore.totalReadLatency'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

        WriteKBps = $_.Group | where{$_.MetricId -eq 'datastore.totalWriteLatency.average'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

    }

}

Should be fine?

[LucD]

It should be ok with a few changes

$VMName = "MyVM"

$stat = 'datastore.totalReadLatency.average','datastore.totalWriteLatency.average'

$dsTab = @{}

$vm = Get-VM -Name $VMName

Get-Datastore -RelatedObject $vm | where{$_.Type -eq 'VMFS'} |

ForEach-Object -Process {

    $dsTab.Add($_.ExtensionData.Info.Vmfs.Uuid,$_.Name)

}

Get-Stat -Entity $vm -Stat $stat -Realtime -MaxSamples 1 |

Group-Object -Property Instance |

ForEach-Object -Process {

    [pscustomobject]@{

        DS = $dsTab[$_.Name]

        Interval = $_.Group[0].IntervalSecs

        ReadLatencyMs = $_.Group | where{$_.MetricId -eq 'datastore.totalReadLatency.average'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

        WriteLatencyMs = $_.Group | where{$_.MetricId -eq 'datastore.totalWriteLatency.average'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

    }

}

[SCharchouf]

Thanks LucD

it's possible to get naa.XXXX information?

as the script collect only datastore information?

the main idea is to identify which virtuell disk has a high I/O, for now really no high latency identified with script output everything seems to be OK

[LucD]

If you are looking at latency per harddisk, you would perhaps use something like the following

$VMName = "MyVM"

$stat = 'virtualDisk.totalReadLatency.average','virtualDisk.totalWriteLatency.average'

$vm = Get-VM -Name $VMName

$vDiskTab = @{}

Get-ScsiController -VM vcsa7 -PipelineVariable scsi |

ForEach-Object -Process {

    Get-HardDisk -VM vcsa7 -PipelineVariable hd | where{$_.ExtensionData.ControllerKey -eq $scsi.Key} |

    ForEach-Object -Process {

        $vDiskTab.Add("scsi$($scsi.ExtensionData.BusNumber):$($hd.ExtensionData.UnitNumber)",$hd.Name)

    }

}

Get-Stat -Entity $vm -Stat $stat -Realtime -MaxSamples 1 |

Group-Object -Property Instance |

ForEach-Object -Process {

    [pscustomobject]@{

        VM = $vm.Name

        HardDisk = $vDiskTab[$_.Name]

        Interval = $_.Group[0].IntervalSecs

        ReadLatencyMs = $_.Group | where{$_.MetricId -eq 'virtualDisk.totalReadLatency.average'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

        WriteLatencyMs = $_.Group | where{$_.MetricId -eq 'virtualDisk.totalWriteLatency.average'} | Measure-Object -Property Value -Average | Select -ExpandProperty Average

    }

}