VMware Cloud Community
0v3rc10ck3d
Enthusiast
Enthusiast
Jump to solution

Script to Generate IOPS Report

Hello,

     I've been looking around for a script but I cant seem to find what i'm looking for. I would like something that would survey all VM's across all clusters and datastores (everything) And would generate a CSV report that show the IOPS usage per VM's.

I just want to be able to evaluate which VM's in particular are taxing our storage arrays.

Any help or pointers in the right direction would be appreciated.

Thanks!

VCIX6 - NV | VCAP5 - DCA / DCD / CID | vExpert 2014,2015,2016 | http://www.vcrumbs.com - My Virtualization Blog!
1 Solution

Accepted Solutions
LucD
Leadership
Leadership
Jump to solution

You mean something like this ?

It collects the IOPS for the complete previous day and send the result via email

$metrics = "disk.numberwrite.summation","disk.numberread.summation"
$finish = Get-Date -Hour 0 -Minute 0 -Second 0
$start = $finish.AddDays(-1)
$report = @()

$vms = Get-VM | where {$_.PowerState -eq "PoweredOn"}
$stats = Get-Stat -Stat $metrics -Entity $vms -Start $start -Finish $finish
$interval = $stats[0].IntervalSecs

$lunTab = @{}
foreach($ds in (Get-Datastore -VM $vms | where {$_.Type -eq "VMFS"})){
 
$ds.ExtensionData.Info.Vmfs.Extent | %{
   
$lunTab[$_.DiskName] = $ds.Name
  }
}

$report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{
 
$readStat = $_.Group |
   
where{$_.MetricId -eq "disk.numberread.summation"} |
   
Measure-Object -Property Value -Average -Maximum
 
$writeStat = $_.Group |
   
where{$_.MetricId -eq "disk.numberwrite.summation"} |
   
Measure-Object -Property Value -Average -Maximum
 
New-Object PSObject -Property @{
   
VM = $_.Values[0]
   
Start = $start
   
Finish = $finish
   
Disk = $_.Values[1]
   
IOPSWriteMax = [math]::Round($writeStat.Maximum/$interval,0)
   
IOPSWriteAvg = [math]::Round($writeStat.Average/$interval,0)
   
IOPSReadMax = [math]::Round($readStat.Maximum/$interval,0)
   
IOPSReadAvg = [math]::Round($readStat.Average/$interval,0)
   
Datastore = $lunTab[$_.Values[1]]
  }
}

Send-MailMessage -Subject "IOPS Report" -From lucd@lucd.info `
 
-To lucd@lucd.info -SmtpServer mail.lucd.info `
 
-BodyAsHtml -Body ($report | Select VM,Start,Finish,Disk,Datastore,IOPSWriteAvg,
   
IOPSWriteMax,IOPSReadAvg,IOPSReadMax | ConvertTo-Html | Out-String)


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

View solution in original post

26 Replies
LucD
Leadership
Leadership
Jump to solution

Have you already looked at my Get the maximum IOPS ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0v3rc10ck3d
Enthusiast
Enthusiast
Jump to solution

Wow, thank you very much! That was really helpful.

I plan on setting the minutes for 1440 to show me a daily report of MAX IOPS, is there a way to show the daily average IOPS as well on the next column over? Then export it to CSV, organize by average in decending format and send me an email every morning... Haha. I'm sure I'll be able to string together something using this very helpful script!

VCIX6 - NV | VCAP5 - DCA / DCD / CID | vExpert 2014,2015,2016 | http://www.vcrumbs.com - My Virtualization Blog!
Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

You mean something like this ?

It collects the IOPS for the complete previous day and send the result via email

$metrics = "disk.numberwrite.summation","disk.numberread.summation"
$finish = Get-Date -Hour 0 -Minute 0 -Second 0
$start = $finish.AddDays(-1)
$report = @()

$vms = Get-VM | where {$_.PowerState -eq "PoweredOn"}
$stats = Get-Stat -Stat $metrics -Entity $vms -Start $start -Finish $finish
$interval = $stats[0].IntervalSecs

$lunTab = @{}
foreach($ds in (Get-Datastore -VM $vms | where {$_.Type -eq "VMFS"})){
 
$ds.ExtensionData.Info.Vmfs.Extent | %{
   
$lunTab[$_.DiskName] = $ds.Name
  }
}

$report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{
 
$readStat = $_.Group |
   
where{$_.MetricId -eq "disk.numberread.summation"} |
   
Measure-Object -Property Value -Average -Maximum
 
$writeStat = $_.Group |
   
where{$_.MetricId -eq "disk.numberwrite.summation"} |
   
Measure-Object -Property Value -Average -Maximum
 
New-Object PSObject -Property @{
   
VM = $_.Values[0]
   
Start = $start
   
Finish = $finish
   
Disk = $_.Values[1]
   
IOPSWriteMax = [math]::Round($writeStat.Maximum/$interval,0)
   
IOPSWriteAvg = [math]::Round($writeStat.Average/$interval,0)
   
IOPSReadMax = [math]::Round($readStat.Maximum/$interval,0)
   
IOPSReadAvg = [math]::Round($readStat.Average/$interval,0)
   
Datastore = $lunTab[$_.Values[1]]
  }
}

Send-MailMessage -Subject "IOPS Report" -From lucd@lucd.info `
 
-To lucd@lucd.info -SmtpServer mail.lucd.info `
 
-BodyAsHtml -Body ($report | Select VM,Start,Finish,Disk,Datastore,IOPSWriteAvg,
   
IOPSWriteMax,IOPSReadAvg,IOPSReadMax | ConvertTo-Html | Out-String)


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0v3rc10ck3d
Enthusiast
Enthusiast
Jump to solution

Wow....thanks a lot!

I ran into an issue when I tried to run it. Any ideas?

Cannot index into a null array.

At C:\iopsreport.ps1:8 char:20

+ $interval = $stats[ <<<< 0].IntervalSecs

    + CategoryInfo          : InvalidOperation: (0:Int32) [], RuntimeException

    + FullyQualifiedErrorId : NullArray

VCIX6 - NV | VCAP5 - DCA / DCD / CID | vExpert 2014,2015,2016 | http://www.vcrumbs.com - My Virtualization Blog!
Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

That indicates that no data was returned from the Get-Stat cmdlet.

Were the VM you selected powered on during the interval you selected ?

Is the Statistics Level set to the required level for the interval you selected ? These counters require statistics level 3.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
0v3rc10ck3d
Enthusiast
Enthusiast
Jump to solution

I ran it again and it worked. Thanks!!

Do you know of any way to add the Max read and Max write values? As well as the Average Read and Average Write? Giving IOPSAvg and IOPSMax?

I tried this but I got numbers far higher than I expected.

$metrics = "disk.numberwrite.summation","disk.numberread.summation"

$finish = Get-Date -Hour 0 -Minute 0 -Second 0

$start = $finish.AddDays(-1)

$report = @()

$vms = Get-VM | where {$_.PowerState -eq "PoweredOn"}

$stats = Get-Stat -Stat $metrics -Entity $vms -Start $start -Finish $finish

$interval = $stats[0].IntervalSecs

$lunTab = @{}

foreach($ds in (Get-Datastore -VM $vms | where {$_.Type -eq "VMFS"})){

  $ds.ExtensionData.Info.Vmfs.Extent | %{

    $lunTab[$_.DiskName] = $ds.Name

  }

}

$report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{

  $readStat = $_.Group |

    where{$_.MetricId -eq "disk.numberread.summation"} |

    Measure-Object -Property Value -Average -Maximum

  $writeStat = $_.Group |

    where{$_.MetricId -eq "disk.numberwrite.summation"} |

    Measure-Object -Property Value -Average -Maximum

  New-Object PSObject -Property @{

    VM = $_.Values[0]

    Start = $start

    Finish = $finish

    Disk = $_.Values[1]

    IOPSMax = [math]::Round($writeStat.Maximum+$readStat.Maximum//$interval,0)

    IOPSAvg = [math]::Round($writeStat.Average+$readStat.Average/$interval,0)

    Datastore = $lunTab[$_.Values[1]]

  }

}

Send-MailMessage -Subject "IOPS Report" -From asdf@asdf.com `

  -To me@me.com -SmtpServer mail.mailserver.com `

  -BodyAsHtml -Body ($report | Select VM,Start,Finish,Datastore,IOPSAvg,

    IOPSMax | Sort-Object IOPSAvg -descending | ConvertTo-Html | Out-String)

VCIX6 - NV | VCAP5 - DCA / DCD / CID | vExpert 2014,2015,2016 | http://www.vcrumbs.com - My Virtualization Blog!
Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

I think that this

    IOPSMax = [math]::Round($writeStat.Maximum+$readStat.Maximum//$interval,0)

    IOPSAvg = [math]::Round($writeStat.Average+$readStat.Average/$interval,0)

should be like this

    IOPSMax = [math]::Round(($writeStat.Maximum+$readStat.Maximum)/$interval,0)

    IOPSAvg = [math]::Round(($writeStat.Average+$readStat.Average)/$interval,0)


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
0v3rc10ck3d
Enthusiast
Enthusiast
Jump to solution

Amazing, Thanks for everything!

VCIX6 - NV | VCAP5 - DCA / DCD / CID | vExpert 2014,2015,2016 | http://www.vcrumbs.com - My Virtualization Blog!
Reply
0 Kudos
AlbertWT
Virtuoso
Virtuoso
Jump to solution

Thanks Luc, you are awesome !

/* Please feel free to provide any comments or input you may have. */
Reply
0 Kudos
ggmattingly
Contributor
Contributor
Jump to solution

Hi Everybody,

Great script LucD...

I had some curious results returned on a few VMs.

Basically it looks like the get-stat returned an instance for a datastore that is not actually part of the VM. 

That is the disk ID, naa.60060e801606400000010640023455601 for example, is returned by Get-Stats. This corresponds to datastore PROD-MGT-L001-NL7K, for example,  which I can verify by looking at that datastore's properties, but if I do a get-datastore for the given VM, this datastore is not listed. The VM has no virtual disk on this datastore, there is no iso mapped to the CD-ROM drive from this datastore either. I thought perhaps it was historical data but even if I use the real time switch and get the stats from an hour ago, this disk still shows up as associated with the VMs. Quite puzzling.  Anybody have any ideas on this?

Anyway I also fiddled with adding the VM's home clustername and primary portgroup name to the report using the following

$ClusterTab = @{}

$PortGroupTab = @{}

foreach($vm in $vms ){

      $ClusterTab[$VM.name] = ($vm.vmhost.parent).name

   $PGName = ($vm | Get-NetworkAdapter).networkname

   If ($PGname.GetType().Name -eq "Object[]")  {

  $PortGroupTab[$VM.Name] = $PGName[0]

   }

   else  {

  $PortGroupTab[$VM.Name] = $PGName

   }

}

Need to add the following to the New-Object properties hash list, and add the names to the output select property name list:

Cluster = $ClusterTab[$_.Values[0]]

PortGroup = $PortGroupTab[$_.Values[0]]

Also here is a way to add allocated diskspace aggregated per datastore for each VM.

But this can make the script take a while.

$DiskAllocTab = @{}

Foreach ($vm in $vms)

{

     $DiskAllocTab[$vm.name] = @{}

     foreach($ds in (Get-Datastore -VM $vm | where {$_.Type -eq "VMFS"}))

     {

          $DiskAllocTab[$vm.name][$ds.name] = @{}

          $Alloc = (Get-HardDisk -VM $vm | where {$_.filename -like "*"+$ds.name+"*"} | Measure-Object -Sum CapacityGB).Sum

          if ($Alloc)  {

                 $DiskAllocTab[$vm.name][$ds.name] = $Alloc

          }

          else {

                 $DiskAllocTab[$vm.name][$ds.name] = 0

          }

     }

}

Add the following 3 lines to just before the New-object statement:

$DiskAllocGB = $null

Try {$DiskAllocGB = $diskAllocTab[$_.Values[0]][$lunTab[$_.Values[1]]]} Catch {$DiskAllocGB = 0}

if ($DiskAllocGB -eq $null) {$DiskAllocGB = 0}

Add the following line to the New-object property hash list:

DiskAllocGB = $DiskAllocGB

Then add the property name select statement for output.

DiskAllocGB

Cheers All!

If anybody has any idea why I get that weird anomaly of phantom datastores, drop a line here... I'm really stumped by it.

Garrett

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Does this VM has snapshots ?

If yes, was that datastore perhaps used at the time one of the snapshots was taken ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
ggmattingly
Contributor
Contributor
Jump to solution

That was a good thought... but no... no snapshot.

Check out the attachment. the first command shows the diskname of the datastore in question.

The second shows the datastores attached to the VM in question.

The third shows filtered data gathered by the get-stat command.

Where did it get that instance from?! Pretty weird.

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

Do you by any chance have/had an ISO image mounted on the CD/DVD drive ?

The name seems to indicate it concerns an ISO repository


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
ggmattingly
Contributor
Contributor
Jump to solution

That is mainly what that datastore is used for... mapping ISO files to a VM's CD-ROM. At the current time the VMs in question have no ISO mapped to the CD-ROM. It is likely that they did have an ISO mapped from this datastore in the past. This would be true for many of the VMs we have. Although as far as I can see the disk metrics for Get-stat do not include the CD-ROM drive, I know this because there are VMs in the report that have the CR-ROM mapped to this datastore and the datastore is not listed for that VM in the report.  It is almost like there was a vdisk for the VM on that datastore at one time and a lingering record of it still remains. So odd...  like something is not updated correctly.

Reply
0 Kudos
ggmattingly
Contributor
Contributor
Jump to solution

I'm going to migrate one of the VMs to a different datastore and see if that clears it up.

Reply
0 Kudos
ggmattingly
Contributor
Contributor
Jump to solution

Migrating one of the VMs to a different datastore fixed it for that VM.  I went to one of the other VMs with the issue and downloaded the VMX file and took a look at it. It has a reference to a virtual disk that is on the datastore in question! Funny thing is that when you edit the VM this disk does not show up. When you do a "get-datastore  for that VM the datastore in question does not show up. But obviously, get-stat is processing this reference in the VMX.  Anyway, LucD, thanks for the suggestions...  always good to have another perspective. At least I know why Get-stat is reporting on that datastore disk instance. I need to go have talk with our VMware admins...

Reply
0 Kudos
santoshannie
Contributor
Contributor
Jump to solution

Hi Lucd , 

This is not working failing with same error . 

Reply
0 Kudos
LucD
Leadership
Leadership
Jump to solution

That metric requires Statistics Level 3.

numberwrite.jpg

PS: cross-posting your question doesn't really help.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
Rocket74
Contributor
Contributor
Jump to solution

Hi there @LucD ,

Can I just say thanks for the script and for the comments on this KB formu 

I want to look at changing the date format

from

$finish = Get-Date -Hour 0 -Minute 0 -Second 0
$start = $finish.AddDays(-1)

To  

$metrics = "disk.numberwrite.summation","disk.numberread.summation"
$startdate = Get-Date "31-02-2023"
$finishdate = Get-Date "31-03-2023"
$report = @()

Is this possible, I'm having issues with either the date format or do I need to add hhmmss as well.

Thanks

Reply
0 Kudos