VMware Cloud Community
stony007_de
Enthusiast
Enthusiast

After KB-1008130 Patch losing Heatbeat after snapshot

Hi Guys,

On friday i was installing the I/O Error Fix Patch " KB-1008130. The install was fine and without problems. Now i got the following problem.

If i start a Backup via vcb, the VMWareTools runs and the VMWare-"Tools Status=ToolsOk".

The snapshot is done and the VMwareTools get the state "Not running"

so the backup cannot work successfull! If i start the vmware tools manually the same effect appears.

any ideas?

stony007_de

Reply
0 Kudos
77 Replies
mk75
Contributor
Contributor

Looks like we have one thing common here.

We use Netbackup too.

Going to test tomorrow doing plain VCB backups and see what happen.

/mk75

Reply
0 Kudos
mk75
Contributor
Contributor

Did some tests today using vcbmounter.

Same problem as with Netbackup.

As soon as VC start creating snapshot Vmware Tools change status to "Not running".

However one difference.

vcbmounter successfully finished copying the vmdks which Netbackup fail to do when this error happen.

/mk75

Reply
0 Kudos
EPL
Contributor
Contributor

I'll throw my hat into the mix as well... I'm getting the same result, and I'm running NetBackup 6.5.3. Snapshot error 156.

Since we don't have a huge amount of VM's, and since it doesn't look like its affecting ALL of them, just some of them (which is strange) I found that simply logging into the vm, and logging out will cause the tool to start responding. The a restart of the Netbackup job will succeed.

Where's the bug fix VMWARE?

UPDATE: I called into Vmware's support and was informed that they are aware of the issue and have escallated to RnD for a fix. No ETA yet. Internal Vmware Bug ID: 374389

Message was edited by: EPL

Reply
0 Kudos
VMegalodon
Contributor
Contributor

So i finally recieved an answer from VMWare Support:

"A fix has been identified for patch 12, but I cannot confirm if this will be the exact release vehicle"

II am not sure what this means but i will update to VC 2.5 Update 4 (released 23 feb) somewhere his week, hope that solves it.

Reply
0 Kudos
Glenn77
Contributor
Contributor

Lame workaround for windows VM's, but it's easier than logging into each server.... from a workstation have a batch file to restart vmtools service

sc
<vmname> stop vmtools

sc
<vmname> start vmtools

Argh! can't show slashes with the post

sc double backslash <servername> stop vmtools

sc double backslash <servername> start vmtools

Now just copy and paste, and put in servernames for all your vm's that you need visible for backup by netbackup and schedule it to run every day until they come out with the patch.

Reply
0 Kudos
Jan_MS
Contributor
Contributor

Today, I got this answer from VMware Support:

"The issue that you reported will be resolved in ESX3.5 Update 4 As well as the ESX 3.5 patch bundle (both due in March)."

Reply
0 Kudos
DonCarter
Contributor
Contributor

I was told that the patch will be released in March also.

The reason some backups fail when the status show "not running"is because they're using VCB. I have the same VMware Tools issues with the bug except the backups still run okay, but we don't use VCB. We use NFS with a Netapp appliance and call the backups via Snap Manager for Virtual Infrastructer (SMVI).

I guess using that strategy paid off in this scenario...sheww! Sorry to the others having to suffer through this until the patch is released...

Reply
0 Kudos
monderick
Enthusiast
Enthusiast

upgraded to VirtualCenter U4 this morning, and the problem still exists.

netbackup calls out VCB, completes the backup, and vmtools goes to 'not running' causing subsequent jobs to fail.

also, is there some trick to getting crontab to run on ESX?

is doesn't want to run the 'service mgmt-vmware restart' command.

Reply
0 Kudos
turkina
Contributor
Contributor

00 8 * * * root service mgmt-vmware restart

Simply add this line to /etc/crontab. Change the "8" to whatever hour you want in 24 hour format.

VCP3/4/5, VCAP5-DCA
Reply
0 Kudos
monderick
Enthusiast
Enthusiast

00 8 * * * root service mgmt-vmware restart

Simply add this line to /etc/crontab. Change the "8" to whatever hour you want in 24 hour format.

works great, thanks. i was doing it incorrectly.

Reply
0 Kudos
VMegalodon
Contributor
Contributor

hmm, i edited the cromtab file but nothing happened. Does the host need a reboot?

Reply
0 Kudos
private00
Enthusiast
Enthusiast

Any Patch available? ?:|

Same Problem after patching the ESX- Hosts totay Smiley Sad After running a Backup with VCB on some VMs the VMTools going to "Not running".

Reply
0 Kudos
monderick
Enthusiast
Enthusiast

hmm, i edited the cromtab file but nothing happened. Does the host need a reboot?

i had to restart the crond service

Reply
0 Kudos
DenisRossi
Contributor
Contributor

Same problem, after backup by vcb a lot of vm report a "tools not running" state.

I don't want to restart mgmt service by cron because this causes a "host disconnect" alarm and sametime DRS problem

We are waiting for a ESX-patch that we hope will be avaible soon.

Reply
0 Kudos
stony007_de
Enthusiast
Enthusiast

if you won't restart the mgmt-vmware service, you can restart the vmware-tools in the vm! in my workaround, i restart the service in the post-exec command of each backup job.

Reply
0 Kudos
DenisRossi
Contributor
Contributor

we have hundred of vm....

it is a very good solution, but it is very expensive...

Reply
0 Kudos
stony007_de
Enthusiast
Enthusiast

ok, in my case we have aprox. 120 VMs! and it works.

yes but with some configuration pain!

Reply
0 Kudos
private00
Enthusiast
Enthusiast

Hi,

answer from VMware support:

This is a known issue with VCB, and our engineers are working on the resolution.

The possible workaround to this issue at the moment are:

1. Restart mgmt-vmware service immediately once the backup job is done. This will bring the tools status to OK. One can write a cron job to do it periodically.

2. Log in and Log out OR simply log out (if already logged in) from the VM. This will cause the tools status to go "OK" if it was showing as "Not running" before.

3. Use VCBMounter to look for VM name or UUID rather than VM IP. VM IP will only work when the status of tools is "OK", but VM name and UUID will work even if the tools status shows as "Not running".

The patch will be available as soon as our engineers will find the root cause of the issue and write appropriate patch.

This could take weeks or months.

Chris

Reply
0 Kudos
Redmption
Contributor
Contributor

Hi!

We have the same problem with a customer of ours.

VC says VMware tools are 'not running' while the are.

Because of this VCB Failes!

I have opened a SR at VMware, and they answerd me that it will be fixed in update 4 of ESX.

We can't wait until then for this to be solved, so they say to use the Crontab workaround.

But because of our monitoring, if we do this a support engingeer will be notified everyday when the crontab runs, and VC looses his connection.

So for us this isn't a solution either.

Has anyone heard something of a temporary fix for this problem?

Reply
0 Kudos
cf13
Enthusiast
Enthusiast

Read all posts in this thread. There are some other solutions - for example restarting vmware tools service after backup.

Reply
0 Kudos