Firstly - this is really a message to LucD (who I know from previous assistance tends to prowl this forum ) , just figured I'd share it, as it's all usefull info and any insight LucD has will no doubt be usefull for all.
Thanks Tony.
1) Yes, you can call Get-Stat with multiple metrics, the downside is that the results will be mixed and you will have to extract the correct metric/value objects. What is even better, you can also pass all the VMs in one call to Get-Stat.
2) The Export-Csv doesn't allow you to specify the order of the columns in the .csv file I'm afraid.
The alternative is to write/construct the .csv file yourself.
Capture each line as you like it in a string and then write this string to a file.
This is the updated script
# Main Variables
$sVCentre = "VC IP"
$sVCUser = "administrator@domain"
$sVCPwd = "password"
$strVMWildCard = "*"
$strCSVName = "Stats-AvgVMDiskWriteStatsWorkingday"
$strCSVLocation = "c:\" $metrics = "virtualDisk.totalWriteLatency.average","virtualDisk.totalReadLatency.average", "virtualDisk.numberReadAveraged.average","virtualDisk.numberWriteAveraged.average", "virtualDisk.read.average","virtualDisk.write.average"
# define the start and finish times for a working day. $today9am = (Get-Date -Hour 9 -Minute 0 -Second 0) $today5pm = (Get-Date -Hour 17 -Minute 0 -Second 0) $intStartDay = -1
$intEndDay = -1
## Begin Script
#Connect to VC
Connect-VIServer $sVCentre -User $sVCUser -Password $sVCPwd -ea silentlycontinue
$arrVMs = Get-VM | where-object {$_.Name -like $strVMWildCard} $stats = Get-Stat -Entity $arrVMs -Stat $metrics -Start $today9am.AddDays($intStartDay) -Finish $today5pm.AddDays($intEndDay)
# group the data and collate the stats into averages for the day.
$groups = $stats | Group-Object -Property {$_.Entity, $_.MetricId, $_.Instance} $report = $groups | % { New-Object PSObject -Property @{ Description = $_.Group[0].Description Entity = $_.Group[0].Entity EntityId = $_.Group[0].EntityId Instance = $_.Group[0].Instance MetricId = $_.Group[0].MetricId Timestamp = $_.Group[0].Timestamp.Date.AddHours($_.Group[0].Timestamp.Hour) Unit = $_.Group[0].Unit Value = [math]::Round(($_.Group | Measure-Object -Property Value -Average).Average, 2) } } #Exporting the report to a CSV file.
$strCSVSuffix = (get-date).toString('yyyyMMddhhmm') $strCSVFile = $strCSVLocation + $strCSVName + "_" + $strCSVSuffix + ".csv"$report | Export-Csv $strCSVfile -NoTypeInformation -UseCulture
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Thanks Tony.
1) Yes, you can call Get-Stat with multiple metrics, the downside is that the results will be mixed and you will have to extract the correct metric/value objects. What is even better, you can also pass all the VMs in one call to Get-Stat.
2) The Export-Csv doesn't allow you to specify the order of the columns in the .csv file I'm afraid.
The alternative is to write/construct the .csv file yourself.
Capture each line as you like it in a string and then write this string to a file.
This is the updated script
# Main Variables
$sVCentre = "VC IP"
$sVCUser = "administrator@domain"
$sVCPwd = "password"
$strVMWildCard = "*"
$strCSVName = "Stats-AvgVMDiskWriteStatsWorkingday"
$strCSVLocation = "c:\" $metrics = "virtualDisk.totalWriteLatency.average","virtualDisk.totalReadLatency.average", "virtualDisk.numberReadAveraged.average","virtualDisk.numberWriteAveraged.average", "virtualDisk.read.average","virtualDisk.write.average"
# define the start and finish times for a working day. $today9am = (Get-Date -Hour 9 -Minute 0 -Second 0) $today5pm = (Get-Date -Hour 17 -Minute 0 -Second 0) $intStartDay = -1
$intEndDay = -1
## Begin Script
#Connect to VC
Connect-VIServer $sVCentre -User $sVCUser -Password $sVCPwd -ea silentlycontinue
$arrVMs = Get-VM | where-object {$_.Name -like $strVMWildCard} $stats = Get-Stat -Entity $arrVMs -Stat $metrics -Start $today9am.AddDays($intStartDay) -Finish $today5pm.AddDays($intEndDay)
# group the data and collate the stats into averages for the day.
$groups = $stats | Group-Object -Property {$_.Entity, $_.MetricId, $_.Instance} $report = $groups | % { New-Object PSObject -Property @{ Description = $_.Group[0].Description Entity = $_.Group[0].Entity EntityId = $_.Group[0].EntityId Instance = $_.Group[0].Instance MetricId = $_.Group[0].MetricId Timestamp = $_.Group[0].Timestamp.Date.AddHours($_.Group[0].Timestamp.Hour) Unit = $_.Group[0].Unit Value = [math]::Round(($_.Group | Measure-Object -Property Value -Average).Average, 2) } } #Exporting the report to a CSV file.
$strCSVSuffix = (get-date).toString('yyyyMMddhhmm') $strCSVFile = $strCSVLocation + $strCSVName + "_" + $strCSVSuffix + ".csv"$report | Export-Csv $strCSVfile -NoTypeInformation -UseCulture
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi LucD,
Yep - once again - spot on.
Now runs a LOT faster , checked the stats and it all seems to work! So quite pleased, thanks once again for the assistance.
TonyG
Hi LucD
Pushing things a little further now and wondering if I can get Datastore stats in the same fashion? - We mainly use NFS these days and therefore - there are no real multipathing issues as a single VMK IP is assigned. Below is a revised piece of code that seems to do the job?? Can you see any pitfalls in this - is theresomething obvious I'm missing in this approach?
I do get results and they seem valid with one exception - the NFS DataStore is refrenced by an InstanceID, and I'm unsure how to resolve this back to the actual DataStore name? - I've been through the Step-3 pages you wrote but this seems to be all SCSI Adapaters and the like, not sure they are valid for NFS Data?
Any assistance to resolve the NFS Instance name to the DataStore name would be helpfull.
THanks again
TG
A little more info.
I've managed to find a reference to the 'instance' GUID given to each datastore in the Web based Managed Object Reference on vCenter.
It's in :
Hi TG,
That is indeed the unique identifier for a datastore.
In the following script I first create a hash table with all these identifiers and the corresponding datastorename.
When you interprete the statistics, you use the Instance as a key into the hash table and it will return the datastorename.
# Main Variables
$sVCentre = "VCIP"
$sVCUser = "administrator@domain"
$sVCPwd = "password"
$strHostWildCard = "name*"
$strCSVName = "Stats-AvgHostDiskWriteStatsWorkingDay"
$strCSVLocation = "c:\"
$arrMetrics = "datastore.numberReadAveraged.average","datastore.numberWriteAveraged.average", "datastore.read.average","datastore.write.average","datastore.totalReadLatency.average", "datastore.totalWriteLatency.average"
# define the start and finish times for a working day.
$today9am = (Get-Date -Hour 9 -Minute 0 -Second 0) $today5pm = (Get-Date -Hour 17 -Minute 0 -Second 0) $intStartDay = -1$intEndDay = -1
## Begin Script
#Connect to VC
Connect-VIServer $sVCentre -User $sVCUser -Password $sVCPwd -ea silentlycontinue
$arrHosts = Get-VMHOST | where-object {$_.Name -like $strHostWildCard -and $_.ConnectionState -eq "Connected" }
# Create hash table Instance-DSname
$hashTab = @{} $arrHosts | %{ $_.Extensiondata.Config.FileSystemVolume.MountInfo | %{ $key = $_.MountInfo.Path.Split('/')[-1] if(!$hashTab.ContainsKey($key)){ $hashTab[$key] = $_.Volume.Name } } }
# Get the statistics
$stats = Get-Stat -Entity $arrHosts -Stat $arrMetrics -Start $today9am.AddDays($intStartDay) -Finish $today5pm.AddDays($intEndDay) $groups = $stats | Group-Object -Property {$_.Entity, $_.MetricId, $_.Instance} #$groups = $stats | Group-Object -Property {$_.Timestamp, $_.Instance}
$report = $groups | % { New-Object PSObject -Property @{ Description = $_.Group[0].Description Entity = $_.Group[0].Entity.Name EntityId = $_.Group[0].EntityId DSname = $hashTab[$_.Group[0].Instance] MetricId = $_.Group[0].MetricId Timestamp = $_.Group[0].Timestamp.Date.AddHours($_.Group[0].Timestamp.Hour) Unit = $_.Group[0].Unit Value = [math]::Round(($_.Group | Measure-Object -Property Value -Average).Average, 2) } }
#Exporting the report to a CSV file.
$strCSVSuffix = (get-date).toString('yyyyMMddhhmm') $strCSVFile = $strCSVLocation + $strCSVName + "_" + $strCSVSuffix + ".csv"
Write-Host "Output file written to : $strCSVfile"
$report | Export-Csv $strCSVfile -NoTypeInformation -UseCulture
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Luc ,
Once again – your a god!
Great work and my sincere thanks :smileygrin:
Best regards
Tony Gent
VCP, EMCTA - NAS Specialist
$metrics = "virtualDisk.totalWriteLatency.average","virtualDisk.totalReadLatency.average", "virtualDisk.numberReadAveraged.average","virtualDisk.numberWriteAveraged.average", "virtualDisk.read.average","virtualDisk.write.average"
What statistics level do I need to enable in order to retrieve these metrics?
Using the default statistic levels, I only have these metrics available for my VMs:
PS C:\> get-vm MyVM | Get-StatType
cpu.usage.average
cpu.usagemhz.average
cpu.ready.summation
mem.usage.average
mem.swapinRate.average
mem.swapoutRate.average
mem.vmmemctl.average
mem.consumed.average
mem.overhead.average
disk.usage.average
net.usage.average
sys.uptime.latest
sys.heartbeat.summation
disk.used.latest
disk.used.latest
disk.used.latest
disk.used.latest
disk.used.latest
disk.provisioned.latest
disk.unshared.latest
As you can see, I have no virtualDisk.* counters at all.
PowerCLI Version
----------------
VMware vSphere PowerCLI 4.1 U1 build 332441
---------------
Any advice?
You can check which statistics level you need in the SDK Reference guide.
Go to the PerformanceManager page and select the group of metrics you are interested in.
In this case that is the virtual disk group.
As you can see, these metrics need statistics level 2.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Hi LucD.
First of all thank you for your job. I'm not a big fan of scripting, but with your guides I start to appriciate it :smileygrin:
I was looking for a script that let me take information about disk usage of VMs, and I've tried you script.
When I run it I recieve this error:
Get-Stat : 19/04/2012 13:33:00 Get-Stat The metric counter "virtualdi
sk.totalreadlatency.average" doesn't exist for entity "dns_cl".
At C:\Scripts\Disk_Perf.ps1:23 char:18
+ $stats = Get-Stat <<<< -Entity $arrVMs -Stat $metrics -Start $todayMidnight.
AddDays(-1) -Finish $todayMidnight
+ CategoryInfo : ResourceUnavailable: (virtualdisk.totalreadlaten
cy.average:String) [Get-Stat], VimException
+ FullyQualifiedErrorId : Client20_RuntimeDataServiceImpl_CheckUserMetrics
_MetricDoesntExist,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats
Get-Stat : 19/04/2012 13:33:00 Get-Stat The metric counter "virtualdi
sk.numberreadaveraged.average" doesn't exist for entity "dns_cl".
At C:\Scripts\Disk_Perf.ps1:23 char:18
+ $stats = Get-Stat <<<< -Entity $arrVMs -Stat $metrics -Start $todayMidnight.
AddDays(-1) -Finish $todayMidnight
+ CategoryInfo : ResourceUnavailable: (virtualdisk.numberreadaver
aged.average:String) [Get-Stat], VimException
+ FullyQualifiedErrorId : Client20_RuntimeDataServiceImpl_CheckUserMetrics
_MetricDoesntExist,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats
Get-Stat : 19/04/2012 13:33:00 Get-Stat The metric counter "virtualdi
sk.numberwriteaveraged.average" doesn't exist for entity "dns_cl".
At C:\Scripts\Disk_Perf.ps1:23 char:18
+ $stats = Get-Stat <<<< -Entity $arrVMs -Stat $metrics -Start $todayMidnight.
AddDays(-1) -Finish $todayMidnight
+ CategoryInfo : ResourceUnavailable: (virtualdisk.numberwriteave
raged.average:String) [Get-Stat], VimException
+ FullyQualifiedErrorId : Client20_RuntimeDataServiceImpl_CheckUserMetrics
_MetricDoesntExist,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats
Get-Stat : 19/04/2012 13:33:00 Get-Stat The metric counter "virtualdi
sk.read.average" doesn't exist for entity "dns_cl".
At C:\Scripts\Disk_Perf.ps1:23 char:18
+ $stats = Get-Stat <<<< -Entity $arrVMs -Stat $metrics -Start $todayMidnight.
AddDays(-1) -Finish $todayMidnight
+ CategoryInfo : ResourceUnavailable: (virtualdisk.read.average:S
tring) [Get-Stat], VimException
+ FullyQualifiedErrorId : Client20_RuntimeDataServiceImpl_CheckUserMetrics
_MetricDoesntExist,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats
Get-Stat : 19/04/2012 13:33:00 Get-Stat The metric counter "virtualdi
sk.write.average" doesn't exist for entity "dns_cl".
At C:\Scripts\Disk_Perf.ps1:23 char:18
+ $stats = Get-Stat <<<< -Entity $arrVMs -Stat $metrics -Start $todayMidnight.
AddDays(-1) -Finish $todayMidnight
+ CategoryInfo : ResourceUnavailable: (virtualdisk.write.average:
String) [Get-Stat], VimException
+ FullyQualifiedErrorId : Client20_RuntimeDataServiceImpl_CheckUserMetrics
_MetricDoesntExist,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetViStats
And this is true for all the virtual machines in my vsphere 4.1 cluster...
How come?
What am I missing?
Thank you
It looks as if your Statistics Level for the Historical Interval is not at level 2.
That level is required to have this specific metric.
You can check from the vSphere Client.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
You were right...
I have changed the level to 2.
Now, when I run the script I get this error:
At C:\Scripts\Disk_Perf.ps1:23 char:18
+ $stats = Get-Stat <<<< -Entity $arrVMs -Stat $metrics -Start $todayMidnight.
AddDays(-1) -Finish $todayMidnight
+ CategoryInfo : NotSpecified: (:) [Get-Stat], VimException
+ FullyQualifiedErrorId : Core_BaseCmdlet_UnknownError,VMware.VimAutomatio
n.ViCore.Cmdlets.Commands.GetViStats
Cannot index into a null array.
At C:\Scripts\Disk_Perf.ps1:29 char:32
+ Description = $_.Group[ <<<< 0].Description
+ CategoryInfo : InvalidOperation: (0:Int32) [], RuntimeException
+ FullyQualifiedErrorId : NullArray
Export-Csv : Cannot bind argument to parameter 'InputObject' because it is null
.
At C:\Scripts\Disk_Perf.ps1:42 char:21
+ $report | Export-Csv <<<< $strCSVfile -NoTypeInformation -UseCulture
+ CategoryInfo : InvalidData: (:) [Export-Csv], ParameterBindingV
alidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,M
icrosoft.PowerShell.Commands.ExportCsvCommand
For reference I attach the script I'm running
Thank you LucD
You'll have to wait at least 1 day till the aggregation jobs have collected the data for that interval.
A good way to check if the values are there, is by going to the Performance tab in the vSphere Client and selecting for example The Past Day.
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
I did it and I can see the data in the performance tab of vsphere client.
Actually, looking at the errors, it looks to me like the script can't resolve some variables.
Am I wrong?
Are you sure
Get-VM | where-object {$_.Name -like $strVMWildCard}
this returns any VMs ?
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference
Yep.
one I define the variable $strVMWildCard = "*" the command give to me the list of all the vms in my cluster
Now it works, but there is something I can't understand....
In the report I get this timestamp fot all the metrics of my vms:
19/04/2012 23:00 |
I was thinking I wuold have had a hourly report.
Again, what am I missing?
Thanks
It's THAT Day people! - The day I actually post something back to a forum..!! I Only do it once a year so I hope you find it usefull
Firstly - let me say that in reality - this is almost all LucD's work, just formatted into a way that I could use it. I've also done a fair bit of poking around and reformatting of output etc.
The export to CSV comes one VM at a time to ensure the data does not get HUGE for big environments, and you can choose the number of days taken and the output for CSV has been tuned to make it work with Pivot Tables in Excel.
I've tested on Powershell 2, PowerCLI 5 and vSphere 4 and 5 I think.
If I've missed anything important - sorry - I only do this once a year!
So cut and paste, type in your vCenter details, perhaps limit the wildcard for the VM's and away you go.
And lets all prostrate ourselves at the feet of the mighty LucD and marvel in his glory - without him - my life inparticular would be so much harder! :smileygrin:
Have a good weekend all!
TG
# Name : Stats-VMDiskWeek_v7# Purpose : This script will run through all the VM's on a given ESX host and export the stats to a CVS file.# Version : 7.0# Authors : LucD, VMware Powershell Forums & Tony Gent, SNS Limited.# Global Site Variables# vCenter Login Variables$sVCentre = "<VC IP>" # holds the fqdn or ip of the vcenter server.$sVCUser = "<VCadministratorUser>"# holds the login name used to login to vcenter, local or domain$sVCPwd = "<VCAdminPwd>" # the password for vcenter login in clear txt.$strVMWildCard = "*" # use * for all machine, or use wildcard to limit, eg : "Sol*" for machines begining Sol$strOutputLocation = "<Path for Export eg. E:\" # The default output location for for results, the file name is generated - just provide the path.# for Stats-VMDiskWeek$strStatsOutput = "Stats-VMDisk"$today5pm = (Get-Date -Hour 17 -Minute 0 -Second 0) # define the start time as 5pm YESTERDAY night.# Define the number of days ago to start the scan.#(Ideally -7 or lower. High numbers (-8 etc) returns 30 mins stats)$intStartDay = -1 # stats for 1 days ago# define the number of days ago to stop the scan.#(Ideally -1 to return stats up to 5pm last night. -0 returns stats up to today )$intEndDay = -0 # stats for 1 days ago (yesterday)## -6, -1 returns 6 days worth of data from 5pm 6 days ago until 5pm last night.## -7, -1 returns 7 days worth of data from 5pm 7 days ago until 5pm last night.## -7, -0 returns 7 days worth of stats from 5pm 7 days ago until 2hrs ago.# Create Log Fileswrite-host "Getting log file name"Write-host "Log file is $sLogfile"# Local Variables$arrMetrics = "virtualDisk.totalWriteLatency.average","virtualDisk.totalReadLatency.average","virtualDisk.numberReadAveraged.average","virtualDisk.numberWriteAveraged.average","virtualDisk.read.average","virtualDisk.write.average"## Begin Script#Connect to VCConnect-VIServer $sVCentre -User $sVCUser -Password $sVCPwd -ea silentlycontinue$arrVMs = Get-VM | where-object {$_.Name -like $strVMWildCard -and $_.PowerState -eq "PoweredOn" }$iCountVMs = $arrVMs.Countwrite-host "List of VM's Genarated, $iCountVMs found."# Create single output file to be appended to by all VMs$strCSVSuffix = (get-date).toString('yyyyMMddhhmm')$strCSVFile = $strOutputLocation + $strStatsOutput + "_" + $strCSVSuffix + ".csv"write-host "Output file written to : $strCSVfile"#Itterate through the Array for all VM's and ask for stats and export.foreach($VM in $arrVMs){write-host "Getting VM Stats for VM : $VM"$Report = Get-Stat -Entity $VM -Stat $arrMetrics -Start $today5pm.AddDays($intStartDay) -Finish $today5pm.AddDays($intEndDay)# Export the stats as a CSV file into the object : $csvExportif ($Report -eq $Null){write-host "******** Stats for VM : $VM are shown as NULL ********"}else{$oStatsObjects = @()Foreach ($oEntry in $Report) {$oObject = New-Object PSObject -Property @{#Description = $oEntry.DescriptionEntity = $oEntry.Entity#EntityId = $oEntry.EntityIdInstance = $oEntry.InstanceMetricId = $oEntry.MetricIdTimestamp = ($oentry.Timestamp).toString('dd/MM/yyyy hh:mm')Value = $oEntry.ValueUnit = $oEntry.Unit}$oStatsObjects += $oObject}write-host "Converting to CSV"$csvExport = $oStatsObjects | ConvertTo-Csv -outvariable $csvOut -notypeinformation#Appending the export object to the report CSV file.write-host "Exporting Data for VM : $VM"$csvExport[1..($csvExport.count -1)] | foreach-object {add-content -value $_ -path $strCSVfile}}#Clear the value of $report to recieve the next machines entries.$Report=$null$oStatsObjects=$null}# Disconnect from VIrtual Centerwrite-host "Script Complete : Disconnecting from vCenter."Disconnect-VIServer -Confirm:$Falsewrite-host "vCenter Disconnect Completed. Script Closed."
Hi Tonygent.
Your script is GREAT!
It's almost what I was looking for.
The only thing is that I need the column name in the report.
I tried to get a new script using yours and the one of LucD, but I wan't able to do the right mix... :smileygrin:
I can't understand why the script of LucD give me only the 11:00 timestamp, and the yours didn't create the colum name in the report...
I know it's a stupid thing, but I really what to tray to get what I need...
Thank you all anyway for your help
V.
Try changing this line
$csvExport[1..($csvExport.count -1)] | foreach-object {add-content -value $_ -path $strCSVfile}
into this
$csvExport[0..($csvExport.count -1)] | foreach-object {add-content -value $_ -path $strCSVfile}
Blog: lucd.info Twitter: @LucD22 Co-author PowerCLI Reference