VMware Cloud Community
bingscott
Contributor
Contributor

Downloading a datastore folder using Powershell

I am attempting to download all of the files in a particular datastore to a local USB drive using Powershell.  Some of the files are *.vmdk files.  When an *.vmdk file is downloaded the system splits it into two files - a 1KB descriptor file and a flat vmdx file.  When the GetDatastoreItem command encounters an *.vmdk file, the copy process quits with the following error:

PS C:\> Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbing\Downloads\ESX_NFS_FACTS_SSD\alb-sys-factsprod\Backup\

Copy-DatastoreItem : 9/17/2018 11:20:38 AM VimDatastore Download of file 'https://alb-sys-vcsprd.ad

.cyfd.state.nm.us/folder%2falb-sys-factsprod%2falb-sys-factsprod-e5800be6.vswp?dcPath=ABQ&dsName=ESX

_NFS_FACTS_SSD' failed. Error message: Response status code does not indicate success: 500

(Internal Server Error).

At line:1 char:1

+ Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbin ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Copy-DatastoreItem], VimException

    + FullyQualifiedErrorId : Client20_StorageServiceImpl_DownloadFile_DownloadFailed,VMware.VimAut

   omation.ViCore.Cmdlets.Commands.CopyDatastoreItem

Copy-DatastoreItem : Object reference not set to an instance of an object.

At line:1 char:1

+ Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbin ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Copy-DatastoreItem], NullReferenceException

    + FullyQualifiedErrorId : System.NullReferenceException,VMware.VimAutomation.ViCore.Cmdlets.Com

   mands.CopyDatastoreItem

What am I missing? I used an exact copy of a Copy-DatastoreItem command used by someone else to copy a datastore folder containing *.vmdk files and everything worked perfectly for them.  Perhaps the Copy-DatastoreItem command cannot handle *.vmdk files.

here is my complete Copy-DatastoreItem command:

PS C:\> Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbing\Downloads\ESX_NFS_FACTS_SSD\alb-sys-factsprod\Backup\

Reply
0 Kudos
11 Replies
LucD
Leadership
Leadership

I normally copy a datastore folder like this

$dsName = 'MyDS'

$folder = 'MyVM'


$ds = Get-Datastore -Name $dsName


New-PSDrive -Location $ds -Name DS -PSProvider VimDatastore -Root "\" | Out-Null

Get-ChildItem -Path "DS:/$folder" -Recurse | %{

   Copy-DatastoreItem -Item "DS:/$($_.DatastoreFullPath.Split(' ')[1])" -Destination C:\Temp

}


Remove-PSDrive -Name DS -Confirm:$false


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
bingscott
Contributor
Contributor

LucD,

Thank you... I tried your script/procedure provided above. It almost worked.  It did copy the entire folder; however, it did not complete the downloading of the vmdk files.  Typically, a download of a vmdk file spawns two files - a small 1 KB description file and a vmdk.flat file that contains a complete data file dump.   It seems like the GetDatasourceItem command cannot handle this backup file spawning. The small description file downloaded successfully, but all of my vmdk files are empty. 

It gets this error when it attempts to back up the vmdk file to the vmdk.flat file:

Copy-DatastoreItem : Object reference not set to an instance of an object.

I am performing this backup using an active online VM. Does a VM need to be powered off, before performing a backup?

scott...

Reply
0 Kudos
LucD
Leadership
Leadership

When I run this, I also get 2 files for a VMDK, but the -flat file is copied as well.

Could it be that there has been a timeout?

You could play with the timeout value via the WebOperationTimeoutSeconds on the Set-PowerCLIConfiguration cmdlet.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
bingscott
Contributor
Contributor

The script work flawlessly when the machine was powered off.  All of the files were copied successfully.  The only thing I noticed is that the reported backup file sizes are slightly off from the original file sizes as reported by VCenter.  It could be that the Copy-DataStoreItem command compresses the resulting copied file. 

Reply
0 Kudos
LucD
Leadership
Leadership

Could be, but could also be caused by the blocksize difference on the two sides.
Is it a big difference?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
bingscott
Contributor
Contributor

When I run the same script against a different datastore and folder, the file sizes are identical, but for the datastore that I am working on the sizes are quite different.  The file sizes of the copied/downloaded files are smaller.  The same thing happens when I copy the directory through VCenter.  The sizes for this datastore do not match.  There are several case reported about this in Google.

I do set the timeout value to forever in my download script.

Reply
0 Kudos
LucD
Leadership
Leadership

Could it be that one VMDK is allocated as Thick and the other one as Thin?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
bingscott
Contributor
Contributor

I checked and these machines use Thin Provisioning.  I guess that I could try restoring one of the vmdk files to a different name to see if it restores to the original size. I am not even sure that I can rename when uploading files.

Reply
0 Kudos
LucD
Leadership
Leadership

I suspect you will have to edit the header file, the -flat should be ok.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos
bingscott
Contributor
Contributor

I stand corrected. The file sizes of the backup files are larger than those reported by the VMWare Datastore screen.  When I first looked at the values, I thought that the files sizes of the backups were smaller, but just the opposite is true. All of the associated descriptor files show that the vmdks are ddb.thinProvisioned = "1".

Reply
0 Kudos
LucD
Leadership
Leadership

And does the size of the copied file correspond with the allocation (not what is actually in use)?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

Reply
0 Kudos