Powershell script to list all VMFS datastore that have more than 25 ms latency ?

[AlbertWT]

Hi All,

is it possible using powershell script to list all VMFS datastore in VCenter that is currently above 25 ms highest datastore latency into CSV or excel spreadsheet ?

[LucD]

Try something like this

$msecs = 25
$stat = "datastore.sizeNormalizedDatastoreLatency.average"
$ds = Get-Datastore | where {$_.Type -eq "VMFS"}
$esx = Get-VMHost -Datastore $ds
$dsTab = @{}
$ds | %{
  $dsTab.Add($_.ExtensionData.Info.Vmfs.Uuid,$_.Name)
}
Get-Stat -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |
where {$_.Value -ge $msecs * 1000} |
Select Timestamp,@{N="ESXi";E={$_.Entity.Name}},
  @{N="DS";E={$dsTab[$_.Instance]}},
  @{N="Latency";E={$_.Value/1000}}

Note that the script filters on VMFS datastore, those are the only ones where this counter is available afaik.

[AlbertWT]

Thanks Luc,

where can I put the time indicator to get the start & end time window and also Export-CSV command ?

so for example I need to know which VMFS datastore that are busy during specific time on the weekdays.

it is similar like your solution on this thread: Script to list vm io latency and the datastore the vm is on 🙂

so instead of supplying the VM name, I just need to input the millisecond and the date / time range.

[AlbertWT]

For some reason the script stopped with the error below:

Get-Stat : 5/07/2014 11:33:36 AM    Get-Stat        The metric counter "datastore.sizenormalizeddatastorelatency.average" doesn't exist for entity "ESXi09-PROD.domain.com".   

At line:9 char:1

+ Get-Stat -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($ds ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ResourceUnavailable: (datastore.sizen...latency.average:String) [Get-Stat], VimException

    + FullyQualifiedErrorId : Client20_RuntimeDataServiceImpl_CheckUserMetrics_MetricDoesntExist,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats

how is that error only happens in one particular hosts ?

[LucD]

Might be a version thing.

Which PowerCLI and vSphere version are you running this against ?

[AlbertWT]

Hi Luc,

it is executed using PowerCLI v5.5 R2 (from workstation) against ESXi 5.1u1

and I have also executed the script against  PowerCLI v5.5 R1 from the VCenter server to the same VCenter managed ESXi hosts.

PowerCLI Version

----------------

   VMware vSphere PowerCLI 5.5 Release 1 build 1295336

---------------

Snapin Versions

---------------

   VMWare AutoDeploy PowerCLI Component 5.5 build 1262826

   VMWare ImageBuilder PowerCLI Component 5.5 build 1262826

   VMware License PowerCLI Component 5.5 build 1265954

   VMware VDS PowerCLI Component 5.5 build 1295334

   VMware vSphere PowerCLI Component 5.5 build 1295334

[LucD]

Strange, that counter was introduced in ESXi 4.1, and is available in the Realtime interval.

Do you see it in the list when you do (for any ESXi host that has these datastores connected) ?

Get-StatType -Entity (Get-VMHost MyESX) -Realtime | where {$_ -match "datastore"}

[AlbertWT]

here's the result:

1. datastore.datastoreNormalWriteLatency.latest
2. datastore.siocActiveTimePercentage.average
3. datastore.numberWriteAveraged.average
4. datastore.datastoreWriteIops.latest
5. datastore.sizeNormalizedDatastoreLatency.average
6. datastore.datastoreMaxQueueDepth.latest
7. datastore.datastoreReadOIO.latest
8. datastore.write.average
9. datastore.datastoreReadLoadMetric.latest
10. datastore.numberReadAveraged.average
11. datastore.datastoreReadIops.latest
12. datastore.totalReadLatency.average
13. datastore.datastoreWriteOIO.latest
14. datastore.datastoreVMObservedLatency.latest
15. datastore.datastoreIops.average
16. datastore.read.average
17. datastore.datastoreWriteLoadMetric.latest
18. datastore.maxTotalLatency.latest
19. datastore.datastoreReadBytes.latest
20. datastore.datastoreNormalReadLatency.latest
21. datastore.datastoreWriteBytes.latest
22. datastore.totalWriteLatency.average

[LucD]

The counter seems to be present (line 5).

It could be that for that specific ESXi there no metrics (for whatever reason).

Try adding "-ErrorAction SilentlyContinue" on the Get-Stat cmdlet.

[AlbertWT]

Luc,

Somehow the script doesn't return any result ?

[LucD]

That could mean there are no datastores with a latency equal or higher than 25 msecs.

Try lowering the threshold, and see if that returns any results.

[AlbertWT]

Hm... possibly ? because it was over the weekend.

Is there any way to specify the date time perhaps ?

[LucD]

Not, since the script uses Realtime metrics.

If you set the threshold rather low (for example $msecs = 0.01), do you see some datastores appearing ?

[AlbertWT]

Still the same issue here Luc, even when I set the threshold to 1 it doesn't show any VMFS datastore at all ?

Get-Stat : 8/07/2014 12:50:52 PM    Get-Stat        Stat with instance '5271b38f-04ec6666-fe65-0017a4770c1c' was not found using the specified filter(s).   

At C:\Admin\Temp\getVMIOPS-DSLatency2.ps1:11 char:9

+ Get-Stat <<<<  -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |

    + CategoryInfo          : ObjectNotFound: (:) [Get-Stat], VimException

    + FullyQualifiedErrorId : Core_OutputHelper_WriteNotFoundError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats

Get-Stat : 8/07/2014 12:50:52 PM    Get-Stat        Stat with instance '5303f13e-b1d6e198-975f-0017a4770da2' was not found using the specified filter(s).   

At C:\Admin\Temp\getVMIOPS-DSLatency2.ps1:11 char:9

+ Get-Stat <<<<  -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |

    + CategoryInfo          : ObjectNotFound: (:) [Get-Stat], VimException

    + FullyQualifiedErrorId : Core_OutputHelper_WriteNotFoundError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats

Get-Stat : 8/07/2014 12:50:52 PM    Get-Stat        Stat with instance '52830e98-37640513-95d3-d8d385aaa78a' was not found using the specified filter(s).   

At C:\Admin\Temp\getVMIOPS-DSLatency2.ps1:11 char:9

+ Get-Stat <<<<  -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |

    + CategoryInfo          : ObjectNotFound: (:) [Get-Stat], VimException

    + FullyQualifiedErrorId : Core_OutputHelper_WriteNotFoundError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats

Get-Stat : 8/07/2014 12:50:52 PM    Get-Stat        Stat with instance '52830e64-eb4f3241-f33f-d8d385aaa78a' was not found using the specified filter(s).   

At C:\Admin\Temp\getVMIOPS-DSLatency2.ps1:11 char:9

+ Get-Stat <<<<  -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |

    + CategoryInfo          : ObjectNotFound: (:) [Get-Stat], VimException

    + FullyQualifiedErrorId : Core_OutputHelper_WriteNotFoundError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats

Get-Stat : 8/07/2014 12:50:52 PM    Get-Stat        Stat with instance '5271b3aa-7a5f9dfc-ad1f-0017a4770c1c' was not found using the specified filter(s).   

At C:\Admin\Temp\getVMIOPS-DSLatency2.ps1:11 char:9

+ Get-Stat <<<<  -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |

    + CategoryInfo          : ObjectNotFound: (:) [Get-Stat], VimException

    + FullyQualifiedErrorId : Core_OutputHelper_WriteNotFoundError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats

[LucD]

The error is saying that there is no metric data for those specific LUNs.

Do you see any performance data for these datastores in the vSphere client or Web client ?

[AlbertWT]

Yes i do, from the vSphere console I can view the performance chart Advanced with Realtime or even custom time range.

For the counters: Read latency, Highest Latency and the write latency in Milliseconds.

[LucD]

Ok, then you could try fetching one of those counters (datastore.totalwritelatency.latest)..

Change that in the script, the rest of the script can stay the same.

[AlbertWT]

Luc,

$msecs = 1

$stat = "datastore.totalWriteLatency.average"

$ds = Get-Datastore | where {$_.Type -eq "VMFS"}

$esx = Get-VMHost -Datastore $ds

$dsTab = @{}

$ds | %{

  $dsTab.Add($_.ExtensionData.Info.Vmfs.Uuid,$_.Name)

}

Get-Stat -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |

where {$_.Value -ge $msecs * 1000} |

Select Timestamp,@{N="ESXi";E={$_.Entity.Name}},

  @{N="DS";E={$dsTab[$_.Instance]}},

  @{N="Latency";E={$_.Value/1000}}

Unfortunately it doesn't work still... the script produce the same error message.

Get-Stat : 8/07/2014 5:22:03 PM    Get-Stat        Stat with instance '5271b3aa-7a5f9dfc-ad1f-0017a4770c1c' was not found using the specified filter(s).  

At C:\Admin\Temp\getVMIOPS-DSLatency2.ps1:11 char:9

+ Get-Stat <<<<  -Entity $esx -Realtime -MaxSamples 1 -Stat $stat -Instance ([array]($dsTab.Keys)) |

    + CategoryInfo          : ObjectNotFound: (:) [Get-Stat], VimException

    + FullyQualifiedErrorId : Core_OutputHelper_WriteNotFoundError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats