WestTexan
Contributor
Contributor

Invoke-VMScript "The guest operations agent could not be contacted."

I am often encountering the following error with Invoke-VMScript:

Invoke-VMScript : 2/3/2012 1:20:21 PM    Invoke-VMScript        The guest operations agent could not be contacted.
At C:\Users\Username\Desktop\test_vm_script.ps1:9 char:23
+ $ret = Invoke-VMScript <<<<  -ScriptText $script -VM $vm -GuestUser $username -GuestPassword $password
    + CategoryInfo          : NotSpecified: (:) [Invoke-VMScript], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_RunScriptInGuest_ViError,VMware.VimAutomation.ViCore.Cmdlets
   .Commands.InvokeVmScript

I can verify that VMware tools is installed, running and showing a status of "Running (Current)" in vSphere Client. I can restart the VMware Tools Service from within Windows on the VM and the next attempts at Invoke-VMScript are successful. This will work for several hours, then I receive the error again until restarting the service again. I'm not finding a single web search result for "The guest operations agent could not be contacted", but this problem is pervasive throughout my vSphere 5.0 environment. Any suggestions?

0 Kudos
24 Replies
LucD
Leadership
Leadership

Is there anything in the System or Application eventlog in the guest ?

Do you have the correct VIX version running on the machine from where you run the Invoke-VMScript ?

Is there anything special in that PowerShell script you are sending to the guest OS ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
WestTexan
Contributor
Contributor

I'm seeing 3 events in the Application event log when I restart the VMware Tools Service from VMUpgradeHelper with Event IDs 258, 270 and 271. All start with "The description for Event ID...cannot be found". The 270 event has the VM's MAC address as the Event Data, but the other two have no EventData.

I can reproduce the problem with the following simple script:

Invoke-VMScript -ScriptText "ipconfig" -VM MyTestVM -GuestCredential (Get-Credential)

It fails until I restart the service, then it works again.

I'm not sure how to determine VIX version, but I get the following output from a Get-PowerCLIVersion

PowerCLI Version
----------------
   VMware vSphere PowerCLI 5.0.1 build 581491
---------------
Snapin Versions
---------------
   VMware AutoDeploy PowerCLI Component 5.0 build 544967
   VMware ImageBuilder PowerCLI Component 5.0 build 544967
   VMware License PowerCLI Component 5.0 build 544881
   VMware vSphere PowerCLI Component 5.0 build 581435
0 Kudos
LucD
Leadership
Leadership

You can determine the VIX version by looking at the VixCOM.dll file in %programfiles%\VMware\VMware VIX.

I have a script in my Will Invoke-VMScript work ? post that checks the prerequisites.

Could you try to run the line with the ScriptType parameter and check if that makes a difference ?

Invoke-VMScript -ScriptText "ipconfig" -ScriptType Bat -VM MyTestVM -GuestCredential (Get-Credential)

The default ScriptType is Powershell, but it seems you are using only DOS commands, so there should be no need to use Powershell.

The eventlog entries created by VMware Tools are somewhat of a mystery. I was never able to find a decent overview of the eventids and what they mean.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
WestTexan
Contributor
Contributor

My VixCOM.dll shows File version 1.10.0.12331, Product version e.x.p build-255297

I added the "-ScriptType Bat" with no change in behavior. I also tested with the 32-Bit PowerCLI with the same results. I tried your Test-InvokeVMScript with the following results:

OK                          : True
VM                          : MyTestVM
PoweredOn                   : True
X86Engine                   : True
ToolsInstalled              : True
Port902Open                 : True
FolderReadAccess            : True
PrivilegeConsoleInteraction : True
SupportedOS                 : True
FQDNorIPConnection          : True

0 Kudos
LucD
Leadership
Leadership

Looks like all the prereqs are there.

Perhaps the VIX logging (see KB1019561) contains some info that could be useful.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
WestTexan
Contributor
Contributor

I'm seeing the following in the vmware.log file in the VM folder on the datastore:

2012-02-11T10:16:17.035Z| vmx| GuestRpcSendTimedOut: message to toolbox-dnd timed out.
0 Kudos
edgrigson
Enthusiast
Enthusiast

I've seen this when running against a VM that's on an older host, ie you're running PowerCLI v5 on the client, the VMtools are up to date but the underlying host is vSphere4 not vSphere5. If so you'll need to run the script in PowerCLI v4 - see my post about the compatibility issue here;

http://www.vexperienced.co.uk/?p=1730

Regards,

Ed

0 Kudos
LucD
Leadership
Leadership

You're probably right Ed.

I have to incorporate that KB in my script.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
Konflikt
Enthusiast
Enthusiast

Unfortunetely I have the same issue when using PowerCLI: Invoke-VMScript "The guest operations agent could not be contacted.":


Invoke-VMScript : 2012.06.05. 20:12:52    Invoke-VMScript        The guest operations agent could not be contacted.
At C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\RunBatOnVMs.ps1:41 char:18
+         Invoke-VMScript <<<<  -ScriptText $script -VM $vm.name -HostUser $HostUser -HostPassword $password -GuestUser $GuestUser -GuestPassword $pass
word -ScriptType Bat -Confirm:$false
    + CategoryInfo          : NotSpecified: (:) [Invoke-VMScript], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_DeleteFileInGuest_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript

Invoke-VMScript : 2012.06.05. 20:12:52    Invoke-VMScript        The guest operations agent could not be contacted.
At C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI\Scripts\RunBatOnVMs.ps1:41 char:18
+         Invoke-VMScript <<<<  -ScriptText $script -VM $vm.name -HostUser $HostUser -HostPassword $password -GuestUser $GuestUser -GuestPassword $pass
word -ScriptType Bat -Confirm:$false
    + CategoryInfo          : NotSpecified: (:) [Invoke-VMScript], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_GetProcessOutputInGuest_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript

System: vCenter5.0, ESXI hosts: 5.0.0, 702118, all of them. the VMs hardware version is 8, and the VMware tools is running&up-to-date.

The VM is WinXP, the problem occurred on lots of WinXP VMs.

PowerCLI is running on the vCenter, which is running on a Win2008 R2 x64 server. But i am using the PowerCLI 32 bit.

VIX version: 1.10.0.12331, bulid 255297

Have you guys any idea, what could be the problem?

Thank you in advance,

Andras

0 Kudos
WestTexan
Contributor
Contributor

Kinflict, are your using AVVI backups with your VMware environment. I am still working on this in a support case with VMware, but we found that if I excluded a VM from backup (Backup Exec 2010 R2 in my case), I could go for several days without encountering this error. For machines that were backed up, the error could be reproducted within a day. I have found that the problem is cleared up temporarily by a restart of VMware tools insided the VM or a vMotion of the VM to another host. I'll post here if/when we ever reach a resolution with VMware support.

0 Kudos
LucD
Leadership
Leadership

It's most probably not limited to Backup Exec, I have been seeing the occasional "GuestRpcSendTimedOut: message to toolbox-dnd timed out." in some of our VMs. And we use the CommVault VSA agent to backup VMs.

I would be very much interested in knowing the resolution of the open case you have.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
Konflikt
Enthusiast
Enthusiast

Thanks for your qucik reply.

I am not using any backup tool/system on these VMs, because these VMs (over 100 winxp VMs) are for testing propuses.

But without PowerCLI, the management of such a lot VMs would much harder, and would cost much additional time. in some cases would be a nightmare.

Workarounds: restarting the VMware tools is good, but not so comfortable. same with vMotion.

what i have done with Invoke-VMScript: runs several small bat scripts inside the VMs. But i am using these scenario since month, and i have never met this issue before. interesting.

edit: typos.

0 Kudos
dmadden
VMware Employee
VMware Employee

Konflikt, I'm the engineer that is working with WestTexan.  Have you opened a support request with VMware?  Can you provide me the case number?

Also, to answer the question about "GuestRpcSendTimedOut: message to toolbox-dnd timed out".  That's a very generic error message.  It can occur in a variety of situations such as high CPU load in the VM, stunned VM (e.g. vmotion) or unresponsive VM.

0 Kudos
LucD
Leadership
Leadership

If "GuestRpcSendTimedOut: message to toolbox-dnd timed out" is generic, any pointers what to look for to actually find the cause of the problem ?


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
dmadden
VMware Employee
VMware Employee

For "GuestRpcSendTimedOut" or the initial problem description of this thread?

If you're curious why you're receiving "GuestRpcSendTimedOut", then I would:

a) examine the performance of the VM at the time of the errors (performance charts in vcenter or guest OS tools such as perfmon or top)

b) Enable vmware tools logging (KB 1017256), reproduce the error and examine the log

c) Check the ESX server logs at the time of the error for any VM operations (hostd/vmkernel/messages)

For the problem in the initial problem description (guest operations agent could not be contacted), we've requested a variety of different logs. The investigation is still ongoing so I can't say with certainty what exact logs to take a look at since the root cause has not been determined.

0 Kudos
Konflikt
Enthusiast
Enthusiast

Thank you for your reply dmadden. Unfortunately my vmware account is not connected to our vSphere licences, so i cannot open a support request. Is there any possibiliy to open a support request without this, or i have to connect with my management, and get that account which has the licences?

Thank you in advance.

0 Kudos
dmadden
VMware Employee
VMware Employee

I would need a support request to be filed.  However, if you can get your management to open the support request but provide your contact details, then I can work with you to troubleshoot the issue.  Thanks.

0 Kudos
LucD
Leadership
Leadership

Thanks for the info.

I'll follow those steps next time I see the message again.


Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

0 Kudos
Konflikt
Enthusiast
Enthusiast

Dear dmadden,

I have opened a support request about this issue with our main VMware account.

The issue deatils are the following: Support Request Confirmation Number: 12183217306. I can use that account to upload any other informations. But please regarding this support request please use my email address in the feature for contact. (you can see it at the end of the reqaust support description)

Thank you in advance.

0 Kudos