VMware Cloud Community
WestTexan
Contributor
Contributor

Invoke-VMScript "The guest operations agent could not be contacted."

I am often encountering the following error with Invoke-VMScript:

Invoke-VMScript : 2/3/2012 1:20:21 PM    Invoke-VMScript        The guest operations agent could not be contacted.
At C:\Users\Username\Desktop\test_vm_script.ps1:9 char:23
+ $ret = Invoke-VMScript <<<<  -ScriptText $script -VM $vm -GuestUser $username -GuestPassword $password
    + CategoryInfo          : NotSpecified: (:) [Invoke-VMScript], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_RunScriptInGuest_ViError,VMware.VimAutomation.ViCore.Cmdlets
   .Commands.InvokeVmScript

I can verify that VMware tools is installed, running and showing a status of "Running (Current)" in vSphere Client. I can restart the VMware Tools Service from within Windows on the VM and the next attempts at Invoke-VMScript are successful. This will work for several hours, then I receive the error again until restarting the service again. I'm not finding a single web search result for "The guest operations agent could not be contacted", but this problem is pervasive throughout my vSphere 5.0 environment. Any suggestions?

Reply
0 Kudos
24 Replies
MKguy
Virtuoso
Virtuoso

I'm not sure how otheres fared with their support requests in the meantime, but I just tracked the issue down by coincidence to the ESXi-side Reload VM operation that Backup Exec triggers on each VM backup too.

The following procedure is consistently reproduceable:

1. Confirm Invoke-VMScript works currently (restart VMware Tools):

Invoke-VMScript -ScriptText 'ipconfig' -VM $vm -GuestCredential $creds

--> Is being executed OK

2. On the ESXi shell of the host running the VM (I actually don't know any other way to trigger the Reload-VM task manually), get the VM ID (or VMX path) with vim-cmd vmsvc/getallvms | grep -i [MyVMName]


3. Now reload the VM via vim-cmd vmsvc/reload [ID]

The vmware.log of the VM will now record the following:

2012-11-19T15:56:59.760Z| vmx| VmdbPipeStreamsOvlError Couldn't read: OVL_STATUS_EOF, (11) Resource temporarily unavailable.
2012-11-19T15:56:59.760Z| vmx| VmdbCnxDisconnect: Disconnect: closed pipe for pub cnx '/db/connection/#6/' (-32)
2012-11-19T15:56:59.760Z| vmx| VmdbDbRemoveCnx: Removing Cnx from Db for '/db/connection/#6/'
2012-11-19T15:56:59.771Z| vmx| SOCKET 2 (78) recv detected client closed connection
2012-11-19T15:56:59.771Z| vmx| Vix: [1889886 mainDispatch.c:2984]: VMAutomation: Connection Error (4) on connection 1.
2012-11-19T15:56:59.818Z| vmx| VmdbAddConnection: cnxPath=/db/connection/#b/, cnxIx=3
2012-11-19T15:56:59.866Z| vmx| Vix: [1889886 vigorCommands.c:940]: VigorHotPlugManagerEndBatchCommandCallback: vmdbErr = -24

4. Try running the Invoke-VMScript command again and it will fail with "The guest operations agent could not be contacted":

Invoke-VMScript -ScriptText 'ipconfig' -VM $vm -GuestCredential $creds

Invoke-VMScript : 19.11.2012 16:57:06    Invoke-VMScript        The guest operations agent could not be contacted.
Bei Zeile:1 Zeichen:16
+ Invoke-VMScript <<<<  -ScriptText 'ipconfig' -VM $vm -GuestCredential $creds
    + CategoryInfo          : NotSpecified: (:) [Invoke-VMScript], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_RunScriptInGuest_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript

5. Restart the VMware Tools Windows service in the VM and Invoke-VMScript will work again until the next reload. Vmware.log messages during tools restart:

2012-11-19T15:57:41.261Z| vcpu-0| TOOLS autoupgrade protocol version 0
2012-11-19T15:57:41.261Z| vcpu-0| TOOLS ToolsCapabilityGuestTempDirectory received 0
2012-11-19T15:57:41.265Z| vcpu-0| GuestRpc: Reinitializing Channel 0(toolbox)
2012-11-19T15:57:41.265Z| vcpu-0| GuestRpc: Channel 0 reinitialized.
2012-11-19T15:57:43.180Z| vcpu-0| GuestRpc: Channel 0, guest application toolbox.
2012-11-19T15:57:43.186Z| vcpu-0| TOOLS ToolsCapabilityGuestTempDirectory received 1 C:\Windows\TEMP\vmware-SYSTEM
2012-11-19T15:57:43.201Z| vcpu-0| TOOLS autoupgrade protocol version 2
2012-11-19T15:57:43.202Z| vcpu-0| TOOLS ToolsCapabilityGuestConfDirectory received C:\ProgramData\VMware\VMware Tools
2012-11-19T15:57:43.203Z| vcpu-0| TOOLS Received tools.set.version rpc call, version = 8389.
2012-11-19T15:57:43.203Z| vcpu-0| ToolsSetVersionWork did nothing; new tools version (8389) matches old Tools version
2012-11-19T15:57:43.204Z| vcpu-0| TOOLS unified loop capability requested by 'toolbox'; now sending options via TCLO
2012-11-19T15:57:43.205Z| vcpu-0| Guest: toolbox: Version: build-652272
2012-11-19T16:00:28.567Z| vcpu-0| TOOLS unified loop capability requested by 'toolbox-ui'; now sending options via TCLO
2012-11-19T16:00:28.567Z| vcpu-0| GuestRpc: Channel 6, guest application toolbox-ui.

I confirmed this behavior for Windows 2003 R2 x86 and Windows 2008 R2 VMs in the following environment:

VMware vSphere PowerCLI 5.0.1 build 581491

VMware ESXi 5.0.0 Update 1 build-821926

VMware Tools  8.6.5.11852 (build-652272) (the 5.0 U1 tools) as well as 9.0.0.15210 (build-782409) (the 5.1 GA tools)

Seeing how the ESXi reload function triggers this issue, it's not Symantec's fault (for once).

-- http://alpacapowered.wordpress.com
Reply
0 Kudos
MKguy
Virtuoso
Virtuoso

For the record, I am also able to reproduce this issue exactly the same way on ESXi 5.1 / vCenter 5.1 / PowerCLI 5.1.

It affects Linux guests too.

-- http://alpacapowered.wordpress.com
Reply
0 Kudos
admin
Immortal
Immortal

Hi,

if you observe the following lines in the hostd.log you're running in an issue which will be fixed in vSphere 5.0 U2 and 5.1 U1 (Ref.: PR850673).

TS [XXX info 'vm:/vmfs/volumes/<UUID>/<VM>/<VM>.vmx' opID=XXX] CheckVMStateForGuestOperation: GuestOps are not ready.
TS [XXX info 'Default' opID=XXX] AdapterServer caught exception: vim.fault.GuestOperationsUnavailable

Best Regards,

Tobias Sutor

Reply
0 Kudos
dmc
Contributor
Contributor

This issue is resolved in 5.0 U2 and 5.1 U1 and documented in the following link:

VMware KB: VMware ESXi 5.1 Patch Image Profile ESXi-5.1.0-20130402001-no-tools

Reply
0 Kudos
assafk
Contributor
Contributor

the same issue happened to me.

i've logged into the VM and restarted the VMtools service

and the problem was resolved

Reply
0 Kudos