Script to list vm io latency and the datastore the vm is on

[jlaco]

Hello.  We have a vsphere 5.0 environment and we are experiencing some heavy io latency.  I have been looking for a powercli script that will get the io latency for each vm and get the datastore name it currently resides on.  We access our storage over fiber.  I'm trying to get a good overview of io latency in one nice view in a csv.  I found what might be a good base at https://communities.vmware.com/thread/304827?start=0&tstart=0 but I'm not quite sure how to get the datastore name into the array and I think it is written for nfs storage anyway.  Thanks in advance for any info\advice!

[LucD]

Try something like this.

$vmName = "VM*"

$stat = "datastore.totalReadLatency.average","datastore.totalWriteLatency.average"
$entity = Get-VM -Name $vmName
$start = (Get-Date).AddHours(-1)

$dsTab = @{}
Get-Datastore | Where {$_.Type -eq "VMFS"} | %{
  $dsTab.Add($_.ExtensionData.Info.Vmfs.Uuid,$_.Name)
}

Get-Stat -Entity $entity -Stat $stat -Start $start |
where {$_.Instance -ne ""} |
Group-Object -Property {$_.Entity.Name},Instance | %{
  New-Object PSObject -Property @{
    VM = $_.Values[0]
    Datastore = $dsTab[$_.Values[1]]
    Start = $start
    Unit = $_.Group[0].Unit
    ReadLatencyAvg = [math]::Round(($_.Group | 
      where {$_.MetricId -eq "datastore.totalReadLatency.average"} |
      Measure-Object -Property Value -Average |
      Select -ExpandProperty Average),2)
    WriteLatencyAvg = [math]::Round(($_.Group | 
      where {$_.MetricId -eq "datastore.totalWriteLatency.average"} |
      Measure-Object -Property Value -Average |
      Select -ExpandProperty Average),2)
  }
}

It will give the average read/write latency over the last hour for each VM you specified in the $vmName variable.

For each VM it will return data for all the datastores on which the VM has files.

[jlaco]

Thanks for the quick response LucD!  I'll check it out and let you know how it goes.

[jlaco]

I ran the script using just one vm to try it out but got this error...

If I want to run this for all VMs in a vCenter do I just use the * in the $vmName?  And can I just throw an "| Export-Csv" at the end to have it write the results out to a csv?  Thanks a bunch for your help!

[LucD]

Could it be that you have multiple connections open ?

How many objects does

$global:defaultVIServers

return ?

Another possibility is that you have datastores with identical UUIDs, which normally shouldn't happen.

Try this adapted version, it will list the duplicate UUIDs.

$vmName = "VM*"

$stat = "datastore.totalReadLatency.average","datastore.totalWriteLatency.average"
$entity = Get-VM -Name $vmName
$start = (Get-Date).AddHours(-1)

$dsTab = @{}
Get-Datastore | Where {$_.Type -eq "VMFS"} | %{
  $key = $_.ExtensionData.Info.Vmfs.Uuid
  if(!$dsTab.ContainsKey($key)){
    $dsTab.Add($key,$_.Name)
  }
  else{
    "Datastore $($_.Name) with UUID $key already in hash table"
  }
}

Get-Stat -Entity $entity -Stat $stat -Start $start |
where {$_.Instance -ne ""} |
Group-Object -Property {$_.Entity.Name},Instance | %{
  New-Object PSObject -Property @{
    VM = $_.Values[0]
    Datastore = $dsTab[$_.Values[1]]
    Start = $start
    Unit = $_.Group[0].Unit
    ReadLatencyAvg = [math]::Round(($_.Group | 
      where {$_.MetricId -eq "datastore.totalReadLatency.average"} |
      Measure-Object -Property Value -Average |
      Select -ExpandProperty Average),2)
    WriteLatencyAvg = [math]::Round(($_.Group | 
      where {$_.MetricId -eq "datastore.totalWriteLatency.average"} |
      Measure-Object -Property Value -Average |
      Select -ExpandProperty Average),2)
  }
}

[jlaco]

The script seems to be working in spite of the error.  I found out that I can put what ever vm query I want in $vmName and all vms get processed fine and the "| Export-Csv" at the end works to write out the results.  It looks like the latency numbers are for the datastore the vm resides on, which is great.  How would I go about tapping into a couple vm stats like mainly the vm latency and the vm average reads\writes?  That would be AWESOME!

I thought I would mention also that we have vCOps and I was looking to see if there was an easy way to see this info in one pane of glass but didn't see a way to do it.  But I was reading about using powercli scripts to create nice scheduled reports in vCOps so I'll be checking that out in the near future too.

Thanks LucD!

[jlaco]

Just seen your reply...  I'll try what you said and post back. 

[jlaco]

I just used your newer script and it is saying that multiple datastores have duplicate uuid's.  If it's not supposed to happen then I assume it could cause some sort of negative impact.  Do you know if there are any adverse affects to duplicate datastore uuids and how this could happen?

[LucD]

Duplicate datastore UUID can have several causes, see Managing Duplicate VMFS Datastores.

Is any of the mentioned causes applicable to your environment  ?

You can get a complete listing of the datastores and their UUID like this.

That might help in trying to determine where the duplicates are coming from.

Get-Datastore | Where {$_.Type -eq "VMFS"} | 
Select Name,@{N="UUID";E={$_.ExtensionData.Info.Vmfs.Uuid}} |
Sort-Object -Property UUID,Name

[jlaco]

Thanks for the script, duplicate uuid info and sharing your knowledge\experience LucD.  You have been way more than helpful!  As I'm looking through the script csv output of the VMs and their datastore IO latency, I'm thinking it would be perfect if I had the vm host name, vm average io latency and vm average iops as well.  If you already have this then AWESOME!  If not, then I'll see what I can put together and post it back.  This would give me a great overview of what's happening from an io perspective in our vmware environment and good information showing the latency issue is with our virtual environment or elsewhere. 

[LucD · Accepted solution]

Try the following version, it includes the average read/write latency for the VM and the average IOPS for the VM.

Since the CSV has a row per datastore, the values for the VM are repeated.

I also added the hostname

$vmName = "VM*"

$stat = "datastore.totalReadLatency.average","datastore.totalWriteLatency.average",
  "datastore.numberReadAveraged.average","datastore.numberWriteAveraged.average"
$entity = Get-VM -Name $vmName
$start = (Get-Date).AddHours(-1)

$dsTab = @{}
Get-Datastore | Where {$_.Type -eq "VMFS"} | %{
  $key = $_.ExtensionData.Info.Vmfs.Uuid
  if(!$dsTab.ContainsKey($key)){
    $dsTab.Add($key,$_.Name)
  }
  else{
    "Datastore $($_.Name) with UUID $key already in hash table"
  }
}

Get-Stat -Entity $entity -Stat $stat -Start $start |
Group-Object -Property {$_.Entity.Name} | %{
  $vmName = $_.Values[0]
  $VMReadLatency = $_.Group |
    where {$_.MetricId -eq "datastore.totalReadLatency.average"} |
    Measure-Object -Property Value -Average |
    Select -ExpandProperty Average
  $VMWriteLatency = $_.Group |
    where {$_.MetricId -eq "datastore.totalWriteLatency.average"} |
    Measure-Object -Property Value -Average |
    Select -ExpandProperty Average
  $VMReadIOPSAverage = $_.Group |
    where {$_.MetricId -eq "datastore.numberReadAveraged.average"} |
    Measure-Object -Property Value -Average |
    Select -ExpandProperty Average
  $VMWriteIOPSAverage = $_.Group |
    where {$_.MetricId -eq "datastore.numberWriteAveraged.average"} |
    Measure-Object -Property Value -Average |
    Select -ExpandProperty Average
  $_.Group | Group-Object -Property Instance | %{
    New-Object PSObject -Property @{
      VM = $vmName
      Host = $_.Group[0].Entity.Host.Name
      Datastore = $dsTab[$($_.Values[0])]
      Start = $start
      DSReadLatencyAvg = [math]::Round(($_.Group | 
          where {$_.MetricId -eq "datastore.totalReadLatency.average"} |
          Measure-Object -Property Value -Average |
          Select -ExpandProperty Average),2)
      DSWriteLatencyAvg = [math]::Round(($_.Group | 
          where {$_.MetricId -eq "datastore.totalWriteLatency.average"} |
          Measure-Object -Property Value -Average |
          Select -ExpandProperty Average),2)
      VMReadLatencyAvg = [math]::Round($VMReadLatency,2)
      VMWriteLatencyAvg = [math]::Round($VMWriteLatency,2)
      VMReadIOPSAvg = [math]::Round($VMReadIOPSAverage,2)
      VMWriteIOPSAvg = [math]::Round($VMWriteIOPSAverage,2)
    }
  }
} | Export-Csv c:\report.csv -NoTypeInformation -UseCulture

[jlaco]

Wow.  Thanks a ton for this script LucD.  It will give me a great overview of our iops activity and help to track down the cause of some latency we are experiencing.  You rock!

[jlaco]

Hi LucD.  I ran the script to check it out and the values for the vm and datastore latencies are the same.  The vm and datastore latency should usually be different right?  I found one instance where the variables were mismatched, VMWriteLatencyAvg = [math]::Round($VMReadLatency,2) and changed that so that's all good now.  Also, if I choose to report back 1 hour, shouldn't each average datastore latency value for that period of time and datastore be the same?  It changes for each VM row.

Does this make sense?  Thanks again for all your help!

[LucD]

Yes, that was a typo, I also corrected the code above.

I guess the VM and datastore latency could be the same if your VM is located on only 1 datastore.

Do you see big differences in the values over a 1 hour period ?

[LucD]

I just did a few tests, and indeed the VM and the DS latency numbers should be the same, provided the VM uses only 1 datastore.

If the VM has harddisks over multiple datastores, the numbers will not be the same.

[jlaco]

Cool thanks for checking.

Here is some output from the script when I run it with more than one vm using $vmName = get-vm.  This is from before I updated the VMWriteLatencyAvg value in the code.

Vms with their varying IO demands spread across multiple hosts but utilizing one datastore.  For a given time period, the average datastore latency should be the same while per vm disk

latency can vary as a result of different io demands right?  Maybe I'm thinking about this wrong though.

[LucD]

Afaik these values are captured per VM, also the DS values.

The DS values are not the total latency for all the VMs that access that datastore.

In the PerformanceManager , under the Datastore counters, you'll notice that the metrics are collected per virtualmachine and per hostsystem.

If you want the total latency for the datastore, you would have to collect the metrics for all virtualmachines that access the datastore, and do the summation

[jlaco]

Alright then that explains it.  Thanks for all the info and help LucD!  Take care

[warkaj]

When I ran this it came back with this as "VMTurbo" is a OVA (Linux box) for monitoring VMWare. It stopped because it couldn't gather information from that particular VM. Is there any way to force it to bypass errors like that and leave them empty?  

[LucD]

Yes, add the parameter "-ErrorAction SIlentlyContinue" to the cmdlet.