Need help with vSphere PowerCLI script to report on all Thin Provisioned DataStores

[jwest1]

Hello I am having trouble writing a script to list used, total % provisioned space and free space for all data stores using the vSphere PowerCLI (formerly VI toolkit). I'd basically like to know what space is being used versus what has been provisioned out on individual vmdk files.

[harkamal]

Tricky. Which storage are you using by the way, 3PAR ??

One option is to get vm's in a data store and sum all vmdk files, but that would be a slow script as it will have to loop through each. Not sure if API's got a direct way of getting to distinguish thin and thick provisioning.

[DougBaer]

To make sure I'm getting your goal, you are talking about thin provisioning on a per-VMDK basis from the ESX side, correct?

I think we can cobble something together that will walk through the VMs and print out the % requested vs. committed space.

Within the API, Uncommitted means: "Total additional storage space, in bytes, potentially used by all virtual machines on this datastore."

For my datastores with all thick disks, Uncommitted is 0. When I have a Thin provisioned disks, that number appears to be the difference between the allocated space and the requested space.

The following may help get you started:

$BYTES_TO_GB = 1024*1024*1024
$MB_TO_GB = 1024
foreach ($ds in Get-Datastore) {
  $dsv = $ds | Get-View
  $dsv.RefreshDatastore()
  $dsv.RefreshDatastoreStorageInfo()
  if ($dsv.Summary.accessible -and $dsv.Capability.PerFileThinProvisioningSupported) {
    $ds.Name + " CapacityGB=" + [int]($ds.CapacityMB / $MB_TO_GB) + " FreeGB=" + `
      [int]($ds.FreeSpaceMB / $MB_TO_GB) + " UncommittedGB=" + [int]($dsv.summary.uncommitted / $BYTES_TO_GB)
  }
}

Message was edited by: DougBaer to add the code

[DougBaer]

Try this as well, from the VM perspective:

$VCServer = "vcenter_server"
$VC = Connect-VIServer $VCServer -User username -Password password

$report = @() 
$allvms = Get-VM
foreach ($vm in $allvms) { 
	$vmview = $vm | Get-View
	foreach($disk in $vmview.Storage.PerDatastoreUsage){ 
   		$dsview = (Get-View $disk.Datastore)
		$dsview.RefreshDatastoreStorageInfo()

		$row = "" | select VMNAME, DATASTORE, VMSIZE_GB, VMUSED_GB, PERCENT
		$row.VMNAME = $vmview.Config.Name				
		$row.DATASTORE = $dsview.Name
		$row.VMSIZE_GB = (($disk.Committed+$disk.Uncommitted)/1024/1024/1024)
		$row.VMUSED_GB = (($disk.Committed)/1024/1024/1024)
		$row.PERCENT = [int](($row.VMUSED_GB / $row.VMSIZE_GB)*100)
		$report += $row 
   } 
} 
$report | Export-Csv "C:\vm_ds.csv" -NoTypeInformation

Disconnect-VIServer $VC -Confirm:$False

It should give you a CSV file with the information on a per-VM basis. VMSIZE_GB is the amount allocated to the VM and VMUSED_GB is the amount of space actually used by that VM's VMDK files. For VMs with VMDKs on more than one datastore, you will have two lines that will need to be added together to get the complete picture. This does not take into account snapshot space or swap space.

I think you could aggregate the data from this report -- sort by Datastore name and get close to what you need.

[jwest1]

That's exactly what I was after and then some. Having not only datastore uncommitted space, but that of the vm will be very useful to be able to report out on. Just out of curiousity, what do you use as an upperlimit for uncommitted freespace? We are thinking of using 70% of space being used on a datastore for saying ok, no more uncommitted space to be doled out to new machines.

One other item to note when running the datastore script is that a number of datastores are reporting uncommitted as being 0. These are thick datastores. So I am wondering why the logic check is not working:

if ($dsv.Summary.accessible -and $dsv.Capability.PerFileThinProvisioningSupported) {

Many thanks again.

[DougBaer]

From what I understand, the PerFileThinProvisioningSupported flag is set if the datastore is CAPABLE of handling thin provisioned VMDKs.

That doesn't mean you have to be doing it

[tpukanecz]

Great script! The problem I was trying to solve was to see if we were dangerously over-provisioned in terms of disk space. We had alot of thin-provisioned VMs and there was concern that if all those VMs started to use the allocated space we would crash the farm. So I built on your script to create this one:

## Powershell script to show the amount of disk space used for all VMs and ammount of disk space allocated
##	on all datastores.  
## Trying to solve the problem of determining not just how much free space is available on the datastores
##	but how much is actually allocated.  Because you can use thin-provisioning in vSphere 4 we need to
##	monitor how much space is actually allocated and potentitially how much we are over provisioned
## Pinched from code in the VMWare communities @ http://communities.vmware.com/message/1355848
##

$VCServer = read-host "VCenter server to query"
#$VC = Connect-VIServer $VCServer -User username -Password password
$VC = Connect-VIServer $VCServer

## Our VMFS datastores use a standard naming convention.  This differentiates them from local storage and
##	templates.  Change the -Name parameter to match your naming conventions
$VMFS = Get-Datastore -refresh -Name "LUN???_VMFS_DS" | Sort-Object $_.Name

$report = @() 								# blank data structure
$allvms = Get-VM | Sort-Object $_.Name		# I likes to sorts my datas
foreach ($vm in $allvms) { 
	$vmview = $vm | Get-View				# get-view to see details of the FM
	foreach($disk in $vmview.Storage.PerDatastoreUsage){ 	# Disks for the VM
   		$dsview = (Get-View $disk.Datastore)				# Datastore used by the disks
		$dsview.RefreshDatastoreStorageInfo()				# refresh to get the latest data
		
		$vmview.Config.Name						# Echo to the screen to show progress

		$row = "" | select VMNAME, DATASTORE, VMSIZE_MB, VMUSED_MB, PERCENT	# blank row
		$row.VMNAME = $vmview.Config.Name					# Add the data to the row
		$row.DATASTORE = $dsview.Name
		$row.VMSIZE_MB = (($disk.Committed+$disk.Uncommitted)/1024/1024)
		$row.VMUSED_MB = (($disk.Committed)/1024/1024)
		$row.PERCENT = [int](($row.VMUSED_MB / $row.VMSIZE_MB)*100)
		$report += $row 													# Add the row to the structure
   } 
} 
$report | Export-Csv "C:\temp\vm_ds.csv" -NoTypeInformation	# dump the report to .csv

$DSReport = @()					# blank data structure for datastore information
foreach ($LUN in $VMFS) {		
	$VMSizeSum = 0				# We will sum the data from the previous report for this LUN
	$LUN.Name					# Echo to screen to show progress.
	
	foreach ($row in $report) {	# Generate sum for this LUN
		if ($row.DATASTORE -eq $LUN.Name) {$VMSizeSum += $row.VMSIZE_MB}
	}
								# Create a blank row and add data to it.
	$DSRow = "" | select Datastore_Name,Capacity_MB,  FreeSpace_MB, Allocated_MB, Unallocated_MB
	$DSRow.Datastore_Name = $LUN.Name
	$DSRow.Capacity_MB = $LUN.CapacityMB
	$DSRow.FreeSpace_MB = $LUN.FreeSpaceMB
	$DSRow.Allocated_MB = [int]$VMSizeSum
	$DSRow.Unallocated_MB = $LUN.CapacityMB - [int]$VMSizeSum	# NB that if we have overallocated disk										# space this will be a negative number
	$DSReport += $DSRow			# add the row to the structure.
}	

$DSReport | Export-Csv "C:\temp\Datastores.csv" -NoTypeInformation	# dump report to .csv

[aerdman]

Hey,

Great scripts.

I'm trying to find the "uncommitted" space on each datastore, by looking at the DatastoreSummary.uncommitted property. I was wondering if anyone has noticed issues trying to get this information when directly querying an ESX server vs vCenter. I find that when using ESX directly, I need to execute RefreshDatastoreStorageInfo() twice before I get up-to-date information.

Also, I'd like to avoid calling RefreshDatastoreStorageInfo() manually, if possible, as the refresh may take some time (per datastore). The documentation says that "The server periodically updates this value" (DatastoreSummary.uncommitted property). Does anyone know how "periodically" this happens and/or if it is configurable?

Thanks,

Ariel

[emptyglass]

Doug, my name is John. I ran this script against my vcenter server and I got this error. I notice I dont get the datastore name or valid size, used, free, or percentage. Can you help?

error in powercli:

you cannot call a method on a null-valued expression.

at c:\program files\...\vm_datastore.ps1:15 char:45

+     $dsview.refreshdatastorestorageinfo <<<<< ()

     + categoryinfo     : invallidoperation: (refreshdatastorestorageinfo:string) [],  runtimeexception

     + fullyqualifiederrorid: invokemethodonnull

attempted to divide by zero.

at c:\program files\...\vm_datastore.ps1:21 char:49

+     $row.percent = [int] (($row.VMUSED_GB / <<<< $row.VMSIZE_GB)*100)

     + catergoryinfo: notspecified: (:) [], runtimeexception

     +fullyqualifiederrorid: runtimeexception

get-view: cannot validate argument on parameter 'viobject'. the argument is null or empty. supply an argument that is not null or empty and then try the command again.

at c:\program files\...\vm_datastore.ps1

+ $dsview = (get-view <<<< $disk.datastore)

+ categoryinfo    : invaliddata: (:) [get-view], parameterbindingvalidationexception

+fullyqualifiederrorid: parameterargumentvalidationerror, vmware.vimautomation.vicore.cmdlets.commands.dotnetinterop.getviview

vm_ds.csv entries:

"vmname", "datastore", "vmsize_gb", "vmused_gb", "percent"

"myvmname", , "0", "0"

[caledfwlch]

An easy way to query all thin provisioned VMDK's is using powerGUI and query the Thin provisioned VMDK, under Best Practice Queries --> Disk Queries
I know that this is an old post, but someone out there might need the info.

Thanks