Hello,
I would like to come up with a script that will monitor all disks on all powered on machines for read and write latency as well as number of writes. I am basically trying to create a tool that will help me better choose which machines might benefit from virtual flash read-cache resources. I have set my statistic level to 2 for all counters except 5min which I set to 4. I am basically wanting to run a script every X mins or hours which will collect these stats per disk and store them in a CSV until i get a a DB setup. I found some code that was written by LucD (i think) that I think should work for this report, but its not working in my environment. The counters are coming back as null. What am I doing wrong? I am able to go into vcenter and look at the performance tabs and see this data for at least the past day, so i know at least some data exists. I have copied my code below:
# Main Variables
$sVCentre = "virtualcenter"
$strVMWildCard = "*"
$strCSVName = "Stats-AvgVMDiskWriteStatsWorkingday"
$strCSVLocation = "c:\temp\"
$metrics = "virtualDisk.totalWriteLatency.average","virtualDisk.totalReadLatency.average",
"virtualDisk.numberReadAveraged.average","virtualDisk.numberWriteAveraged.average",
"virtualDisk.read.average","virtualDisk.write.average"
#define the start and finish times for a working day.
$today9am = (Get-Date -Hour 9 -Minute 0 -Second 0)
$today5pm = (Get-Date -Hour 17 -Minute 0 -Second 0)
$intStartDay = -1
$intEndDay = -1
#Begin Script
#Connect to VC
Connect-VIServer $sVCentre -ea silentlycontinue
$arrVMs = Get-Cluster "Prod 1" | Get-VM | where-object {$_.Name -like $strVMWildCard} | Sort Name
$stats = Get-Stat -Entity $arrVMs -Stat $metrics -Start $today9am.AddDays($intStartDay) -Finish $today5pm.AddDays($intEndDay)
#group the data and collate the stats into averages for the day.
$groups = $stats | Group-Object -Property {$_.Entity, $_.MetricId, $_.Instance}
$report = $groups | % {
New-Object PSObject -Property @{
Description = $_.Group[0].Description
Entity = $_.Group[0].Entity
EntityId = $_.Group[0].EntityId
Instance = $_.Group[0].Instance
MetricId = $_.Group[0].MetricId
Timestamp = $_.Group[0].Timestamp.Date.AddHours($_.Group[0].Timestamp.Hour)
Unit = $_.Group[0].Unit
Value = [math]::Round(($_.Group | Measure-Object -Property Value -Average).Average, 2)
}
}
#Exporting the report to a CSV file.
$strCSVSuffix = (get-date).toString('yyyyMMddhhmm')
$strCSVFile = $strCSVLocation + $strCSVName + "_" + $strCSVSuffix + ".csv"
$report | Export-Csv $strCSVfile -NoTypeInformation -UseCulture
If I add " -IntervalMins 5" to the end of the $stats line, I get data but the script never completes ( i have waited 2 hours) so I am guessing that is not exactly what I want.
Ok I have verified that adding the "-IntervalMins 5" generates the data I want when I do them one at a time, however, the script as its written with that switch has now been running for 8 hours at 50% CPU and still hasn't finished. An example of the data I want to get when I run this one at a time looks like the following. Is there an easier way to get this than to get to 5 minute stats for each counter and then use PowerShell to group them as I have done above?
MetricId | Timestamp | Unit | EntityId | Description | Instance | Value | Entity |
virtualdisk.numberreadaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of read commands issued per second to the virtual disk during the collection interval | scsi0:0 | 0.52 | Ffcobdb |
virtualdisk.numberreadaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of read commands issued per second to the virtual disk during the collection interval | scsi0:1 | 0 | Ffcobdb |
virtualdisk.numberreadaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of read commands issued per second to the virtual disk during the collection interval | scsi0:2 | 429.75 | Ffcobdb |
virtualdisk.numberreadaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of read commands issued per second to the virtual disk during the collection interval | scsi0:3 | 0 | Ffcobdb |
virtualdisk.numberreadaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of read commands issued per second to the virtual disk during the collection interval | scsi0:4 | 0.02 | Ffcobdb |
virtualdisk.numberreadaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of read commands issued per second to the virtual disk during the collection interval | scsi0:5 | 0 | Ffcobdb |
virtualdisk.numberwriteaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of write commands issued per second to the virtual disk during the collection interval | scsi0:0 | 2.92 | Ffcobdb |
virtualdisk.numberwriteaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of write commands issued per second to the virtual disk during the collection interval | scsi0:1 | 0 | Ffcobdb |
virtualdisk.numberwriteaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of write commands issued per second to the virtual disk during the collection interval | scsi0:2 | 0.86 | Ffcobdb |
virtualdisk.numberwriteaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of write commands issued per second to the virtual disk during the collection interval | scsi0:3 | 2.68 | Ffcobdb |
virtualdisk.numberwriteaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of write commands issued per second to the virtual disk during the collection interval | scsi0:4 | 0 | Ffcobdb |
virtualdisk.numberwriteaveraged.average | 3/4/14 23:00 | number | VirtualMachine-vm-37165 | Average number of write commands issued per second to the virtual disk during the collection interval | scsi0:5 | 0.01 | Ffcobdb |
virtualdisk.read.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of reading data from the virtual disk | 26246.61 | Ffcobdb | |
virtualdisk.read.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of reading data from the virtual disk | scsi0:0 | 29.84 | Ffcobdb |
virtualdisk.read.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of reading data from the virtual disk | scsi0:1 | 0 | Ffcobdb |
virtualdisk.read.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of reading data from the virtual disk | scsi0:2 | 26215.78 | Ffcobdb |
virtualdisk.read.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of reading data from the virtual disk | scsi0:3 | 0.02 | Ffcobdb |
virtualdisk.read.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of reading data from the virtual disk | scsi0:4 | 0.07 | Ffcobdb |
virtualdisk.read.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of reading data from the virtual disk | scsi0:5 | 0.22 | Ffcobdb |
virtualdisk.write.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of writing data to the virtual disk | 89.51 | Ffcobdb | |
virtualdisk.write.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of writing data to the virtual disk | scsi0:0 | 63.63 | Ffcobdb |
virtualdisk.write.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of writing data to the virtual disk | scsi0:1 | 0.1 | Ffcobdb |
virtualdisk.write.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of writing data to the virtual disk | scsi0:2 | 10.93 | Ffcobdb |
virtualdisk.write.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of writing data to the virtual disk | scsi0:3 | 2.74 | Ffcobdb |
virtualdisk.write.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of writing data to the virtual disk | scsi0:4 | 6.62 | Ffcobdb |
virtualdisk.write.average | 3/4/14 23:00 | KBps | VirtualMachine-vm-37165 | Rate of writing data to the virtual disk | scsi0:5 | 1.8 | Ffcobdb |
virtualdisk.totalreadlatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a read from the virtual disk takes | scsi0:0 | 10.51 | Ffcobdb |
virtualdisk.totalreadlatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a read from the virtual disk takes | scsi0:1 | 0 | Ffcobdb |
virtualdisk.totalreadlatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a read from the virtual disk takes | scsi0:2 | 9.81 | Ffcobdb |
virtualdisk.totalreadlatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a read from the virtual disk takes | scsi0:3 | 0.08 | Ffcobdb |
virtualdisk.totalreadlatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a read from the virtual disk takes | scsi0:4 | 0.18 | Ffcobdb |
virtualdisk.totalreadlatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a read from the virtual disk takes | scsi0:5 | 1.12 | Ffcobdb |
virtualdisk.totalwritelatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a write to the virtual disk takes | scsi0:0 | 5.14 | Ffcobdb |
virtualdisk.totalwritelatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a write to the virtual disk takes | scsi0:1 | 0.58 | Ffcobdb |
virtualdisk.totalwritelatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a write to the virtual disk takes | scsi0:2 | 1.65 | Ffcobdb |
virtualdisk.totalwritelatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a write to the virtual disk takes | scsi0:3 | 2.04 | Ffcobdb |
virtualdisk.totalwritelatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a write to the virtual disk takes | scsi0:4 | 6.94 | Ffcobdb |
virtualdisk.totalwritelatency.average | 3/4/14 23:00 | millisecond | VirtualMachine-vm-37165 | The average time a write to the virtual disk takes | scsi0:5 | 1.9 | Ffcobdb |
OK so it turns out that the REALLY slow portion of this script turns out to be the grouping portion where i take the stats and group them by entity, disk, and metric. I am trying to derive the average for each of the disk metric types over a 24 hour period. This report pulls up really quickly in the Performance tab in vCenter so surely this value is created somewhere as part of the Daily roll up job. I am just having a hard time getting that value per disk using the Get-Stat command. Help?
There are many factors that might influence the required time to run this script; the number of metrics returned by the Get-Stat cmdlet, the number of VMs for which you run the report.
In the vSphere client you probably get the metrics for 1 specific VM.
There are some options to explore and possibly make this script run a bit faster faster.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Thanks for the advice LucD. I ended up putting the raw stats into a second SQL database that I could properly index and do all of my grouping/reporting out of that database.