topic Re: How to extend/optimize this multi vc datastore inventory script in VMware PowerCLI Discussions

How to extend/optimize this multi vc datastore inventory script

A_S — Fri, 26 Apr 2013 21:24:46 GMT

Hi,

Trying to setup a script for datastore inventory. This is what i have so far.

I would like to extend this datastore inventory script.

      1.) How to add additional properties to this script?
      $row.HostsConnected (i.e. as one joined string esxhost1|esxhost2|esxhost3),
      $row.VMsHosted (i.e. as one joined string VM1|VM2|VM3)
      $row.SiocEnabled (True/False),
      $row.PartofStorageCluster (True/False)
      $row.LUNUUID (yes LUNUUID (fast method there) of Datastore_Name processed during for each loop)

working on

vSphere 5.1.x infra

Powercli 5.1.x

Thanks is advance

# start script
$LogTimeStamp = (Get-Date).ToString("dd-MM-yyyy_HH_mm")
$ScanDate = Get-Date -format "yyyy-MM-dd"
$ScanTime = Get-Date -format "HH:mm"
$ScanDateTimeStamp= Get-Date -format "yyyy-MM-dd HH:mm:00.000"

Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$false
Connect-Viserver "vCenter1.mydomain.corp", "vCenter2.mydomain.corp"

$currDir = Split-path -parent $MyInvocation.MyCommand.Definition
$outputFile = $currDir + "\All_VC" + "_" + $LogTimeStamp + "_datastore_inventory.csv"

if (Test-Path $outputFile)
{
Remove-Item $outputFile
}

cls
$clusterColl = @(Get-Cluster) | Sort-Object Name

$i=0
$b=0
$b = $clusterColl.Count

$report = New-Object Collections.Generic.LinkedList[PSObject]
foreach($dc in get-datacenter) {
foreach($cluster in Get-Cluster -Location $dc) {
    $i++
    write-progress "Please wait....Datastore inventory in progress..." "Cluster ($i of $b) >> $cluster" 1 -perc (($i / $b)*100)

    $VMHost = $cluster | Get-VMHost | select -First 1
write-progress "Selected vSphere server: " "vSphere server >> $VMHost" 2 -perc 100 -parentid 1
$VMHost | Get-DataStore | Where-Object { $_.Name -notlike "*local*"} | Sort-Object Name | get-view | %{
            $row = "" | Select-Object Rec_ID, ScanDate, ScanTime, Infra, vCenter, Datacenter,
            Cluster, Datastore_Name, DeviceID, BlockSizeMB,Vmfsupgradable,Version,Type,
            MaxBlocks,URL, VMs, VMsHosted, Hosts, HostsConnected, Capacity_GB, Used_spaceGB, Free_spaceGB, PercentFree, PercentUsed, LUN_UUID, ScanDateTimeStamp


     $row.Rec_ID=""
            $row.ScanDate = $ScanDate
            $row.ScanTime = $ScanTime
     $row.ScanDateTimeStamp=$ScanDateTimeStamp
     $row.vCenter=$VMHost.extensiondata.Client.ServiceUrl.Split('/')[2].replace(":443","")

Switch ($row.vCenter) {
"vCenter1.mydomain.corp" {$row.Infra="TDA"}
"vCenter1.mydomain.corp" {$row.Infra="PROD"}
}

$row.Datacenter = [string]$dc.Name

$row.Cluster = [string]$Cluster.Name
$row.Datastore_Name = [string]$($_.info.Name)

     if ($_.info.URL.StartsWith("netfs")) {
   # write-host "Exclude NFS datastores"
     }
     else
     {
             $row.Capacity_GB = [int]([math]::round($($_.info.vmfs.Capacity / 1024/1024/1024),2))
      $row.Free_spaceGB = [int]([math]::round($($_.info.Freespace / 1024/1024/1024),2))
      $row.Used_spaceGB = [int]([math]::round($row.Capacity_GB - $row.Free_spaceGB,2))
             $row.PercentFree = [int][math]::round(($row.Free_spaceGB / $row.Capacity_GB * 100),2)
             $row.PercentUsed = [int]100 - $row.PercentFree
      $row.VMs=[int]$($_.vm.count)
      $row.VMsHosted="..."
$row.Hosts=[int]$($_.host.count)
      $row.HostsConnected="..."
$row.DeviceID=[string]$($_.info.Vmfs.Extent[0].DiskName)
      $row.BlockSizeMB=[int]$($_.info.vmfs.BlockSizeMB)
      $row.Version=[string]$($_.info.vmfs.Version)
      $row.Type=[string]$($_.info.vmfs.Type)
      $Row.MaxBlocks=[int]$($_.info.vmfs.MaxBlocks)
      $Row.URL=[string]$($_.info.URL)
      $Row.vmfsupgradable=[bool]$($_.info.vmfs.vmfsupgradable)
$Report.AddLast($row) | Out-Null
     }

}
}

}

$report | Sort-Object Free_spaceGB -Descending | Export-Csv -NoTypeInformation -Path $outputFile

# end script

Re: How to extend/optimize this multi vc datastore inventory script

LucD — Fri, 26 Apr 2013 23:40:24 GMT

These are already 4 of the 5 properties

        $row.HostsConnected= [string]::Join('|',
          (Get-View ($_.ExtensionData.Host | Select -ExpandProperty Key) | %{$_.Name.Split('.')[0]}))
        $row.VMsHosted= [string::Join('|',(Get-View $_.ExtensionData.VM | %{$_.Name}))
        $row.SIOC = $_.StorageIOCOntrolEnabled
        $row.PartofStorageCluster = $_.ParentFolderId -match "StoragePod"

I'm not completely sure what you mean with the LUNUUID (fast method) ?

Is the the CanonicalName for the LUN(s) that form the extents of the datastore ?

Re: How to extend/optimize this multi vc datastore inventory script

A_S — Sat, 27 Apr 2013 09:21:16 GMT

Thank you for taking time (what, it's almost 02:00 a clock in Belgium, don't you sleep ever?).

I like your oneliners in orders to retrieve these properties. They are correct syntatically (if i use them outside of this script) but for some reason they end up with errors inside this script. I suspect that it has to do with the way how the loop for datastore is defined and every time when i try to call something by utilizing .extensiondata then it fails with The argument is null or empty.

I guess this part of a script already does get-view which makes the .extensiondata not usable.

$VMHost | Get-DataStore | Where-Object { $_.Name -notlike "*local*"} | Sort-Object Name | get-view | %{
$row = "" | Select-Object Rec_ID, ScanDate, ScanTime, Infra, vCenter, Datacenter,

The following error message is something i get every time i try to utilize .extensiondata.

Get-View : Cannot validate argument on parameter 'VIObject'. The argument is null or empty. Supply an argument that is not null or empty and then try the command again.
At scriptname.ps1<line #> char:<char#>
+   $row.VMsHosted= [string]::Join("|",(Get-View <<<< $dst.ExtensionData.VM | %{$_.Name}))
    + CategoryInfo          : InvalidData: (:) [Get-View], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutom
   ation.ViCore.Cmdlets.Commands.DotNetInterop.GetVIView

And about LUN uuid:

i'm using this one but it's so slow. I was looking for something faster that might integrate with this script.

function GetLunUUID([string]$vmhost,[string]$dsname) {
$ds = Get-View (Get-View (Get-VMHost -Name $vmhost).ID).ConfigManager.StorageSystem
foreach ($vol in $ds.FileSystemVolumeInfo.MountInfo) {
if ($vol.Volume.Name -eq $dsname) {
   # Write-host "DS Name:`t" $vol.Volume.Name
   # Write-host "VMFS UUID:`t" $vol.Volume.Uuid
   foreach ($volid in $vol.Volume.Extent) {
    foreach ($lun in $ds.StorageDeviceInfo.MultipathInfo.Lun){
     if ($lun.Id -eq $volid.DiskName) {
      break
     }
    }
   }

# Write-Host "LUN Name:`t" $lun.ID
$mid = $lun.ID
foreach ($id in $ds.StorageDeviceInfo.ScsiLun) {
   if ($id.CanonicalName -eq $mid) {
    $uuid = $id.Uuid
    # Write-host "LUN UUID:`t" $uuid
    }
   }
}
}
return $uuid
}

O;-), and i saw you script with datastores view at: http://www.lucd.info/2011/11/14/storage-views-datastores/.

After this i will try to pick up some addtional properties that you offer in that script. Great!

Re: How to extend/optimize this multi vc datastore inventory script

LucD — Sat, 27 Apr 2013 10:12:30 GMT

Lol, sleeping is a waste of time

Ok, didn't catch that you already did the Get-View on the Datastore object.

In that case, just remove the ExtensionData qualifier.

I'll have a look if I can make the LUNUUID any faster.

Re: How to extend/optimize this multi vc datastore inventory script

A_S — Sat, 27 Apr 2013 11:17:36 GMT

Aha, indeed, it works now.:smileyblush:

There is one downside. The execution time now takes 10 times longer.

$row.VMs=[int]$($_.vm.count)
$row.VMsHosted= [string]::Join("|",(Get-View $_.VM| %{$_.Name}))

$row.Hosts=[int]$($_.host.count)
$row.HostsConnected= [string]::Join("|",(Get-View ($_.Host | Select -ExpandProperty Key) | %{$_.Name.Split('.')[0]}))

It seems that these (red) lines retrieve all properties of VM resp. VMHost and than pass only the names back which slows down the total execution time of script.

thanks in advance!

Re: How to extend/optimize this multi vc datastore inventory script

LucD — Sat, 27 Apr 2013 13:23:21 GMT

See if this makes a difference in execution time

$row.VMs=[int]$($_.vm.count)
$row.VMsHosted= [string]::Join("|",(Get-View $_.VM -Property Name| %{$_.Name}))
$row.Hosts=[int]$($_.host.count)
$row.HostsConnected= [string]::Join("|",(Get-View ($_.Host | Select -ExpandProperty Key) -Property Name | %{$_.Name.Split('.')[0]}))

Re: How to extend/optimize this multi vc datastore inventory script

A_S — Sat, 27 Apr 2013 14:36:09 GMT

It works (it's faster). I didn't expect that these 2 extra traversals would be so expensive.

Thank you for helping the community!

Re: How to extend/optimize this multi vc datastore inventory script

LucD — Sun, 28 Apr 2013 09:34:00 GMT

For the LUN UUID you might try the following.

For each VMHost, compile a lookup table containing the datastorename and the datastore's UUID (VMFS only)

...
$VMHost = $cluster | Get-VMHost | select -First 1 
# Create DS-UUID lookup table
$lunUUIDTab = @{}
$storSys = Get-View $esx.ExtensionData.ConfigManager.StorageSystem 
$storSys.FileSystemVolumeInfo.MountInfo | where {$_.Volume.Type -eq "VMFS"} | %{
  $lunUUIDTab.Add($_.Volume.Name,$_.Volume.Uuid)
}

Later in the script fetch the UUID like this

# Fetch the UUID
$row.LUNUUID = $lunUUIDTab[$_.Name]

Re: How to extend/optimize this multi vc datastore inventory script

A_S — Sun, 28 Apr 2013 12:11:23 GMT

It's working!:smileycool:

But it seems that my inventory script (or my approach to get this data) has to be redesigned. I need somehow faster inventory results (nightly scheduled) from customers infra (> 10 VCs, >2 DCs, >100 CLSs, > 600 hosts, > 1000 datastores, >10.000 VMs) and dump them into SQL DB.

The downside: If one of VCs is having problems then the script stops responding.

Every time i contruct something i take the following into consideration

Method 1: Loop through an array of Virtual Centers, one at a time

$VCs= @(“vc_server1”,“vc_server2”) Foreach ($VC in $VCs) { ` Connect-viserver $VC .script... Disconnect-VIServer $VC –confirm:$false }

Method 2: Connect to an array of Virtual Centers, all at the same time (trying to achieve with script)

# Set PowerCLI to MultipleMode
Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$false
connect-VIServer -Server “vc_server1”,“vc_server2”
.script...
Disconnect-VIServer * –confirm:$false

Its seems that i have to go for the third method (because both previous approaches take about 5 hours to get the needed info). Method 2 affects all get-view commands.

Method 3: Trigger every VC at certain scheduled time, let the Powercli script run locally against 'own' VC and dump the results to central SQL DB. i tested and within 1-1,5 hour the results are retrieved. But it's not 'agentless' data collection.

I didn't test it with (4th method) powershell multithreaded option (Add-job). Would that be an alternative for 'agentless data collection' approach?

Anyhow i will (try to) combine this script with additional stuff from your article: http://www.lucd.info/2011/11/14/storage-views-datastores/

in order to get additional info per datastore: [Size_Snapshot] ,[Size_vDisk] ,[Size_Swap] ,[Size_Other] ,[Size_Shared] ,[Multipathing] ,[ConnectivityStatus]. That would make the output complete for different internal usage purposes. The Storage CapMan want's to have it very fast. For me a huge challenge to combine it without degrading in execution time.

Thank you for your help! You're my (and my friend's) Hero!

Re: How to extend/optimize this multi vc datastore inventory script

LucD — Sun, 28 Apr 2013 14:45:10 GMT

In method 1 you can avoid that the script stops when the VC is in trouble by adding the -ErrorAction SilentlyContinue parameter on the Connect-VIServer.

You will have to test yourself if the connected worked or not, and depending on the result retrieve the data or not.

Something like this

$vcName = "MyVC" 

$vc = Connect-VIServer -Server $vcName -ErrorAction SilentlyContinue 
if($vc.IsConnected){
# Collect the date
}
else{
# Connection to VC failed, display error message
  Write-Error "Connection to $($vcName) failed !"
}

Method 2 can indeed take a considerable time in a larger environment.

Here you can optimise the speed by using the Get-View cmdlet and by using the Property parameter.

Like this

Get-View -ViewType HostSystem -Property "Name","VM"

and limit the properties to only the ones you need in the script.

Method 4 would my personal choice, you run/schedule the inventory script from a single station and you use a PowerShell feature to "multi-thread".

But be aware that the Start-Job in combination with a PowerCLI script needs some specific coding in your script.

Re: How to extend/optimize this multi vc datastore inventory script

AmolPatil — Wed, 01 Oct 2014 10:22:30 GMT

Hello LucD,

Hope you are doing good,

Its a very nice script, thanks to A S for making this great script.

Can we make this in reverse direction, I mean based on VM in VC and their respective information like, Host, Datastore ,Cluster and rest properties which are there in script.

I am looking for multiple VCs, is it possible to pull such data from multiple VCs?

Re: How to extend/optimize this multi vc datastore inventory script

AmolPatil — Wed, 08 Oct 2014 04:47:14 GMT

Hello A S & LucD,

With help of your script I modified and build a new where it gives output based on VM and their respective information.

Please have a look and give your feedback if any thing needs to change.

Regards,

Amol Patil