Copying a large file to the guest using Copy-VMGuestFile in a PowerShell script invoked from C# hangs

[icnocop2]

Trying to copy a large file from my local machine to the guest virtual machine from C# hangs.

It works without issues for small files.

Running the PowerShell script directly works without issues.

The file I'm trying to copy to the guest is Windows8.1-KB2919355-x64.msu (690.8 MB) which can be downloaded here:

Download Windows 8.1 Update for x64-based Systems (KB2919355) from Official Microsoft Download Cente...

See the attached example source code with 3 different unit tests (none of which work as expected):

1. CopyFile_UsingVsphereSdk

    Uses the vSphere Management SDK's InitiateFileTransferToGuest crashes the virtual machine.

    See  Copying a large file to the guest causes the virtual machine to hang and eventually crash with error...

2. CopyFile_UsingPowerShellScriptAndRunspace

    Uses PowerShell automation Runspace and Pipeline to invoke a PowerShell script that calls Copy-VMGuestFile.

    Times out with the default WebOperationTimeoutSeconds, otherwise hangs indefinitely (takes longer than 24 hours)

3. CopyFile_UsingPowerShellScript

    Uses PowerShell automation PowerShell.Create() to invoke a PowerShell script that calls Copy-VMGuestFile.

    Times out with the default WebOperationTimeoutSeconds, otherwise hangs indefinitely (takes longer than 24 hours)

VMware vCenter Server 6.0U2+ (6.0.0.3634794)

VMware ESXi Server 6.0U2+ (6.0.0.3825889)

Guest OS: Windows Server 2012 R2

VMware Tools: 10.0.6.3560309

I'm able to reproduce the issue with the attached code sample written in C# targeting .NET 4.5 and references VMware.Vim.dll that comes with VMware-PowerCLI-6.3.0-3737840.

I'm running on Windows 7 Enterprise x64 SP1 and using Visual Studio 2015 Update 3.

1. Save the msu as "C:\KB2919355\Windows8.1-KB2919355-x64.msu" on your local machine.

2. Create the directory "C:\install" in the guest.

3. Run any one of the unit tests

Any ideas?

Thank you!

[LucD]

Did you already check the vmware.log file (in the VM's folder) ?

That might contain further clues for all VMware Tools interactions.

[icnocop2]

Yes.

A snippet of vmware.log was already attached to the referenced thread, Copying a large file to the guest causes the virtual machine to hang and eventually crash with error...

2016-05-29T02:26:44.731Z| vmx| I120: VigorTransportProcessClientPayload: opID=5bdb1be2-e0-5eb0 seq=106189: Receiving GuestOps.InitiateFileTransferToGuest request.

2016-05-29T02:26:44.768Z| vcpu-0| I120: VigorTransport_ServerSendResponse opID=5bdb1be2-e0-5eb0 seq=106189: Completed GuestOperations request.

2016-05-29T02:35:31.867Z| vcpu-0| I120: Unrecoverable memory allocation failure

2016-05-29T02:42:25.368Z| vcpu-0| W110: A core file is available in "/vmfs/volumes/guid/Windows Server 2012 R2/vmx-zdump.003"

2016-05-29T02:42:25.368Z| vcpu-0| W110: Writing monitor corefile "/vmfs/volumes/guid/Windows Server 2012 R2/vmmcores.gz"

2016-05-29T02:42:25.406Z| vcpu-0| W110: CoreDump error line 2160, error Cannot allocate memory

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace:

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[0] 000003ffcaac9550 rip=0000000001aae04e rbx=0000000001aadb30 rbp=000003ffcaac9570 r12=0000000000000000 r13=0000000000000149 r14=000003ffca1cbec0 r15=0000000032052520

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[1] 000003ffcaac9580 rip=00000000015486ea rbx=000000000277b3e8 rbp=000003ffcaac9a70 r12=0000000000000001 r13=0000000000000149 r14=000003ffca1cbec0 r15=0000000032052520

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[2] 000003ffcaac9a80 rip=00000000015381a3 rbx=0000000000010000 rbp=000003ffcaac9a80 r12=00000000026119d0 r13=0000000000000149 r14=000003ffca1cbec0 r15=0000000032052520

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[3] 000003ffcaac9a90 rip=0000000001a0e2f7 rbx=0000000000010000 rbp=000003ffcaac9aa0 r12=00000000026119d0 r13=0000000000000149 r14=000003ffca1cbec0 r15=0000000032052520

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[4] 000003ffcaac9ab0 rip=000000000193f788 rbx=000003ffca1cbec0 rbp=000003ffcaad1bc0 r12=00000000026119d0 r13=0000000000000149 r14=000003ffca1cbec0 r15=0000000032052520

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[5] 000003ffcaad1bd0 rip=0000000001943be3 rbx=000000000286c6d0 rbp=000003ffcaad1c00 r12=00000000026119d0 r13=0000000000000149 r14=000003ffca1cbec0 r15=0000000032052520

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[6] 000003ffcaad1c10 rip=0000000001978de6 rbx=0000000000000000 rbp=000003ffcaad1c50 r12=000000000000012d r13=000000000277b3e8 r14=00000000028651c0 r15=0000000002749020

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[7] 000003ffcaad1c60 rip=0000000001943cf9 rbx=0000000002825600 rbp=000003ffcaad1c60 r12=0000000032057700 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[8] 000003ffcaad1c70 rip=0000000001776e78 rbx=0000000002825600 rbp=000003ffcaad1da0 r12=0000000032057700 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[9] 000003ffcaad1db0 rip=0000000002d23ddc rbx=0000000000000000 rbp=0000000000000000 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[10] 000003ffcaad1ec0 rip=0000000003b6a12d rbx=0000000000000000 rbp=0000000000000000 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000

2016-05-29T02:42:25.406Z| vcpu-0| I120: Backtrace[11] 000003ffcaad1ec8 rip=0000000000000000 rbx=0000000000000000 rbp=0000000000000000 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[0] 000003ffcaac9550 rip=0000000001aae04e in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[1] 000003ffcaac9580 rip=00000000015486ea in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[2] 000003ffcaac9a80 rip=00000000015381a3 in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[3] 000003ffcaac9a90 rip=0000000001a0e2f7 in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[4] 000003ffcaac9ab0 rip=000000000193f788 in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[5] 000003ffcaad1bd0 rip=0000000001943be3 in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[6] 000003ffcaad1c10 rip=0000000001978de6 in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[7] 000003ffcaad1c60 rip=0000000001943cf9 in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[8] 000003ffcaad1c70 rip=0000000001776e78 in function (null) in object /bin/vmx loaded at 00000000013a9000

2016-05-29T02:42:25.406Z| vcpu-0| I120: SymBacktrace[9] 000003ffcaad1db0 rip=0000000002d23ddc in function (null) in object /lib64/libpthread.so.0 loaded at 0000000002d1c000

2016-05-29T02:42:25.407Z| vcpu-0| I120: SymBacktrace[10] 000003ffcaad1ec0 rip=0000000003b6a12d in function clone in object /lib64/libc.so.6 loaded at 0000000003a97000

2016-05-29T02:42:25.407Z| vcpu-0| I120: SymBacktrace[11] 000003ffcaad1ec8 rip=0000000000000000

2016-05-29T02:42:25.407Z| vcpu-0| I120: Msg_Post: Error

2016-05-29T02:42:25.407Z| vcpu-0| I120: [msg.log.error.unrecoverable] VMware ESX unrecoverable error: (vcpu-0)

2016-05-29T02:42:25.407Z| vcpu-0| I120+ Unrecoverable memory allocation failure

[LucD]

Perhaps a stupid question, but did you already try assigning more memory to the target VM?

[icnocop2]

Yes, I increased the VM's RAM from 4 GB to 8 GB but the same error occurred.

When monitoring the VM, its RAM remains steady and close to 1 GB in use (7 GB available) anyway during the entire time.

[LucD]

I tested with a PowerShell script using the InitiateFileTransferToGuest method directly, and that seems to work for me.

I'm curious to see if that gives the same memory error in your environment.

$vmName = 'MyVM'

$vm = Get-VM -Name $vmName

$si = Get-View ServiceInstance

$guestMgr = Get-View -Id $si.Content.GuestOperationsManager

$guestFile = Get-View -Id $guestMgr.FileManager

$auth = New-Object VMware.Vim.NamePasswordAuthentication

$auth.interactiveSession = $false

$auth.username = 'user'

$auth.password = 'password'

$srcFile = 'C:\myfile.dat'

$file = Get-ChildItem -Path $srcFile

$guestFilePath = 'C:\myfile.dat'

$guestFileAttrib = New-Object VMware.Vim.GuestFileAttributes

$size = [long]$file.Length

$overwrite = $false

$resp = $guestFile.InitiateFileTransferToGuest($vm.ExtensionData.MoRef,$auth,$guestFilePath,$guestFileAttrib,$size,$overwrite)

Invoke-WebRequest -Uri $resp -Method Put -Body (Get-Content -Path $srcFile -Raw)

[icnocop2]

Thank you.

Yes, I still get an OutOfMemoryException with your sample code:

Get-Content : Exception of type 'System.OutOfMemoryException' was thrown.

At .\InitiateFileTransfer.ps1:62 char:49

+ Invoke-WebRequest -Uri $resp -Method Put -Body (Get-Content -Path $srcFile ...

+

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (:) [Get-Content], OutOfMemoryException

    + FullyQualifiedErrorId : ProviderContentReadError,Microsoft.PowerShell.Commands.GetContentCommand

[LucD]

Ok, now we know at least on which side the out of memory is occurring.

That means we would need to upload the file in chunks.

Checking how this can be done

[icnocop2]

LucD‌, any updates?

Thank you!

[icnocop2]

Even after upgrading to VMware ESXi 6.0.0 build 4510822, VMware Tools 10.0.9 build 3917699, and VMware-PowerCLI-6.5.0-4624819, the issue still occurs.

If I look at the WebResponse when the error occurs in the C# code as follows:

string response;

WebResponse webResponse = ((WebException)e.Error).Response;

using (StreamReader streamReader = new StreamReader(webResponse.GetResponseStream()))
{
    response = streamReader.ReadToEnd();
}

throw new Exception(string.Format("Upload failed. Response: {0}", response), e.Error);

I get the following response:

<error>
   <errorCode>17</errorCode>
   <errorInfo>The procedure returned an error. A guest may have stopped.</errorInfo>
</error>

And the following exception in e.Error:

System.Net.WebException: The remote server returned an error: (500) Internal Server Error.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.WebClient.GetWebResponse(WebRequest request, IAsyncResult result)
   at System.Net.WebClient.DownloadBitsResponseCallback(IAsyncResult result)

Any work-arounds?