VMware Cloud Community
jlaco
Enthusiast
Enthusiast
Jump to solution

Script to list vm io latency and the datastore the vm is on

Hello.  We have a vsphere 5.0 environment and we are experiencing some heavy io latency.  I have been looking for a powercli script that will get the io latency for each vm and get the datastore name it currently resides on.  We access our storage over fiber.  I'm trying to get a good overview of io latency in one nice view in a csv.  I found what might be a good base at https://communities.vmware.com/thread/304827?start=0&tstart=0 but I'm not quite sure how to get the datastore name into the array and I think it is written for nfs storage anyway.  Thanks in advance for any info\advice!

69 Replies
LucD
Leadership
Leadership
Jump to solution

This line

$start = (Get-Date).AddHours(-1)

into

$start = (Get-Date).AddDays(-20)

or

$start = (Get-Date).AddMonths(-1)

But remember that the interval time in these older metrics is increased due to the aggregation that is done.

See my PowerCLI & vSphere statistics – Part 1 – The basics  post for some background info on this.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

AlbertWT
Virtuoso
Virtuoso
Jump to solution

Thanks LucD, you're the man !

/* Please feel free to provide any comments or input you may have. */
Reply
0 Kudos
Sivapb
Enthusiast
Enthusiast
Jump to solution

Hi LucD,

Script running successfully without Error.But no data in report.

i tried  past two days and all . Still i didnt get any data in report

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Is the statistical data available ?

Do you see the data under Performance in the vSphere client or the Web client ?

Could be that the Statistic Levels in your vCenter do retain these counters that long.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Sivapb
Enthusiast
Enthusiast
Jump to solution

Hi LucD

Yes i can see

i am looking for last two days all VM's disk latency in data store wise in VC server

But i cant get

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Can you attach the script you are currently using ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Sivapb
Enthusiast
Enthusiast
Jump to solution

here the script

======================

$vmName = "*"

$stat = "datastore.totalReadLatency.average","datastore.totalWriteLatency.average",

  "datastore.numberReadAveraged.average","datastore.numberWriteAveraged.average"

$entity = Get-VM -Name $vmName

$start = (Get-Date).AddHours(-1)

$dsTab = @{}

Get-Datastore | Where {$_.Type -eq "VMFS"} | %{

  $key = $_.ExtensionData.Info.Vmfs.Uuid

  if(!$dsTab.ContainsKey($key)){

    $dsTab.Add($key,$_.Name)

  }

  else{

    "Datastore $($_.Name) with UUID $key already in hash table"

  }

}

&{Get-Stat -Entity $entity -Stat $stat -Start $start -ErrorAction SilentlyContinue |

Group-Object -Property {$_.Entity.Name} | %{

  $vmName = $_.Values[0]

  $VMReadLatency = $_.Group |

    where {$_.MetricId -eq "datastore.totalReadLatency.average"} |

    Measure-Object -Property Value -Average |

    Select -ExpandProperty Average

  $VMWriteLatency = $_.Group |

    where {$_.MetricId -eq "datastore.totalWriteLatency.average"} |

    Measure-Object -Property Value -Average |

    Select -ExpandProperty Average

  $VMReadIOPSAverage = $_.Group |

    where {$_.MetricId -eq "datastore.numberReadAveraged.average"} |

    Measure-Object -Property Value -Average |

    Select -ExpandProperty Average

  $VMWriteIOPSAverage = $_.Group |

    where {$_.MetricId -eq "datastore.numberWriteAveraged.average"} |

    Measure-Object -Property Value -Average |

    Select -ExpandProperty Average

  $_.Group | Group-Object -Property Instance | %{

    New-Object PSObject -Property @{

      VM = $vmName

      Host = $_.Group[0].Entity.Host.Name

      Datastore = $dsTab[$($_.Values[0])]

      Start = $start

      DSReadLatencyAvg = [math]::Round(($_.Group |

          where {$_.MetricId -eq "datastore.totalReadLatency.average"} |

          Measure-Object -Property Value -Average |

          Select -ExpandProperty Average),2)

      DSWriteLatencyAvg = [math]::Round(($_.Group |

          where {$_.MetricId -eq "datastore.totalWriteLatency.average"} |

          Measure-Object -Property Value -Average |

          Select -ExpandProperty Average),2)

      VMReadLatencyAvg = [math]::Round($VMReadLatency,2)

      VMWriteLatencyAvg = [math]::Round($VMWriteLatency,2)

      VMReadIOPSAvg = [math]::Round($VMReadIOPSAverage,2)

      VMWriteIOPSAvg = [math]::Round($VMWriteIOPSAverage,2)

    }

  }

}} | Export-Csv c:\report.csv -NoTypeInformation -UseCulture

=====================================================

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Seems to be working for me without a problem.

Which PowerCLI build are you running ?

Do a

Get-PowerCLIVersion

And which vSphere version are you running this against ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Sivapb
Enthusiast
Enthusiast
Jump to solution

Power cli version 5.1 vsphere version 5.1

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

That should work with those versions.

Did you already try with just 1 VM ?

A VM for which you made sure that the data exists.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Sivapb
Enthusiast
Enthusiast
Jump to solution

Yes I tried single VM also where it start $vmName = "*" And executed same script to different VC also. Script running successfully but no data

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Then I'm out of ideas I'm afraid


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Sivapb
Enthusiast
Enthusiast
Jump to solution

Dear LucD, I've tested the script more than 25 VC, Observed following points - Get the output with data exactly for Virtual Center Server 5.0 - Script running successfully but no data in output for Virtual Center Server 5.1 and appliance I'm not sure where should check, All VC setting are same 😞

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Which of the properties in the report are incorrect on VC 5.1 ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
StevenLykins
Contributor
Contributor
Jump to solution

SIvapb,

Please try this.

Change the var start to this

$start = (Get-Date)

and then also change you get-stat statement to this.

&{Get-Stat -Entity $entity -Stat $stat -Realtime -ErrorAction SilentlyContinue |

When looking in Vsphere you may see data there, but its for realtime. In Vsphere if you click on the drop down only see realtime that is all you data for. So when you are looking for old data with this script it comes back with nothing. The changes I have stated will give you 1hour of data from the sec you run it.

Reply
0 Kudos
beharibabu
Contributor
Contributor
Jump to solution

Hi LucD,

Thanks for the script, I have used it and it works.

The result I'm getting is as follows for a VM.

Start           : 2017-09-11 14:49:08

WriteLatencyAvg : 3276773906,3

Unit            : millisecond

ReadLatencyAvg  : 2149940644,19

VM              : *************

Datastore       : ******************

Is the read Latency Value is correct and the unit of value is ms?

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

The value is indeed in ms, but those values you are seeing look way off.

I did notice that sometimes there is an obviously incorrect value returned.
This might be caused by flaws in the statistical sampling process, since these values do look highly unlikely

value-off.jpg

I normally drop those extreme values.

In statistical circles this is called the "exclusion of outliers​", but it is a subject on which opinions differ.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

beharibabu
Contributor
Contributor
Jump to solution

Thanks for your reply..

I have used this script to find the VMs which is having high read latency to assign vFRC.

Unfortunately I don't have any clue now to identify VMs to assign vFRC.

Is there any script or any thoughts on how to identify the VMs for assigning vFRC?

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

I think the method you are using is the way to go (find the high latency vDisks), but throw away the extreme values, to get a more meaningful average.
Btw, this script looks at latency on the datastore level, you might perhaps also look at the vDisk latency values (see 3. Re: measuring_disk_latency_powercli)


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

beharibabu
Contributor
Contributor
Jump to solution

Thanks Again LucD...

I have ran the script and it works.

In order to assign vFRC I have to check the latency when the workload runs on the server.

With this script (Re: measuring_disk_latency_powercli) I can get the Real Time value of latency in vDisk.

Is there a way to get historical data?

Reply
0 Kudos