VMware Cloud Community
statssweden
Contributor
Contributor

"Check and upgrade Tools before each power-on" restarts VM's unexpectedly during a migration from a lesser version host to a ESX 3.5u2!

After upgrading one of my ESX-hosts to 3.5.0,110268 (from 3.5.0,84374) and then migrated VM's back to this host, several of the VM's started to upgrade their VM Tools resulting in unexpected reboots and in some cases OS-freezing in the virtual machine.

The option ”Check and upgrade Tools before each power-on” was activated on all VM's, but since I didn't manually initiate any reboots on the VM's, the upgrade shouldn't have taken place, right?

But after contacting VMware Support with this question, I got the somewhat terrifying answer:

"This is a known issue which will be fixed in a future patch. The issue is caused because when the virtual machine is VMotioned to another ESX host, the tools see a power on event as the virtual machine resumes. Currently the only workaround is to disable the "Check and upgrade Tools before each power-on" option."

The scary part is that there exists no official information regarding this bug:

"Additional information has not been released on this while it is being worked on internally however as I mentioned in my previous mail it will be fixed going forward."

So consider this a warning, don't use the option ”Check and upgrade Tools before each power-on” before VMware has released a patch for this bug.

Tags (1)
Reply
0 Kudos
11 Replies
gdesmo
Enthusiast
Enthusiast

Ouch! I got bit by this issue last night. Only some of my vm's that have this box checked got upgraded/re-booted. Others were fine.. Strange. Any update from support?

I have more hosts to patch tonight. Is there a way to disable this check box without powering down the vm's?

Reply
0 Kudos
henketh
Contributor
Contributor

No, not a word from support! And consequently I'm not sure how this option works in U3, but since they promised to let me know when a solution is available, I guess they haven't fixed it yet..

Yes, you can use VI Toolkit and a script. I believe this is the right script for turning this feature ON:

connect-viserver -server $vm = Get-View (Get-VM VMName).Id $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec $vmConfigSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo $vmConfigSpec.Tools.ToolsUpgradePolicy = "UpgradeAtPowerCycle" $vm.ReconfigVM($vmConfigSpec)

For turning it OFF I believe you should change "UpgradeAtPowerCycle" to "Manual", but I can't confirm this right now since I don't remember it and as I'm at home right now I can't check my script at work.

I believe you can verify the syntax by checking this value in the .vmx-file for a VM. Check the vmx-file for a VM with the "check and upgrade" option turned ON, it should have "UpgradeAtPowerCycle" somewhere, and then check the corresponding value in av .vmx-file belonging to a machine with the feature turned OFF, I'm positive it should read "Manual".

Let me know your results!

Reply
0 Kudos
henketh
Contributor
Contributor

Hello again!

Now I'm at work, and my exact script for the VI Toolkit reads like this, and disables the "Check and upgrade tools at each power-on" on all machines:

(I had to paste this as an image to keep the formatting right)

And in the vmx-file there's a entry that reads:

tools.upgrade.policy = "manual" or "upgradeAtPowerCycle"

Good luck!

Reply
0 Kudos
gdesmo
Enthusiast
Enthusiast

Sorry I am not very verse in scripting. Would I not need the below to connect to my vc server? Your second post looks quite different than your first. I just need to paste your second post in the vi tool kit window?

connect-viserver -server <server> -user <user> -password <pwd>

Thank you

Reply
0 Kudos
henketh
Contributor
Contributor

Sorry, my second post assumed that you already have connected to your vi-server. So use the line

connect-viserver -server <server> -user <user> -password <pwd>

to create the connection

THEN

paste the lines from my second post

and finally press ENTER

Then you can follow the progress of the modification of the VM's in VirtualCenters Recent Tasks pane.

Reply
0 Kudos
gdesmo
Enthusiast
Enthusiast

Hello Again Smiley Happy It errored on me. I think it might be formating? Are there carriage returns after each of your lines in your image?

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; Get-View (Get-VM

VMName).Id $vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec $vmCon

figSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo $vmConfigSpec.Tools.ToolsU

pgradePolicy = "UpgradeAtPowerCycle" $vm.ReconfigVM($vmConfigSpec)

Get-VM : 10/17/2008 12:50:20 PM Get-VM VM with name 'VMName' not foun

d, using the specified filter(s).

At line:1 char:17

+ Get-View (Get-VM &lt;&lt;&lt;&lt; VMName).Id $vmConfigSpec = New-Object VMware.Vim.Virtu

alMachineConfigSpec $vmConfigSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo

$vmConfigSpec.Tools.ToolsUpgradePolicy = "UpgradeAtPowerCycle" $vm.ReconfigVM(

$vmConfigSpec)

You cannot call a method on a null-valued expression.

At line:1 char:229

+ Get-View (Get-VM VMName).Id $vmConfigSpec = New-Object VMware.Vim.VirtualMach

ineConfigSpec $vmConfigSpec.Tools = New-Object VMware.Vim.ToolsConfigInfo $vmCo

nfigSpec.Tools.ToolsUpgradePolicy = "UpgradeAtPowerCycle" $vm.ReconfigVM( &lt;&lt;&lt;&lt;

$vmConfigSpec)

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt;

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; [Reflection.Assem

bly]::LoadWithPartialName("vmware.vim")

GAC Version Location

--- -


-


True v2.0.50727 C:\WINDOWS\assembly\GAC_MSIL\VMware.Vim\1.0.0.0__10980...

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec = N

ew-Object VMware.Vim.VirtualMachineConfigSpec

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec.Too

ls = New-Object VMware.Vim.ToolsConfigInfo

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec.Too

ls.ToolsUpgradePolicy = "UpgradeAtPowerCycle"

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; Get-VM | %{(Get-V

iew $_.ID).ReconfigVM(vmConfigSpec)}

Reply
0 Kudos
henketh
Contributor
Contributor

Hi again!

You don't seem to be using the first line of the script in the image? Reflection.Assembly and so on..?

And then you use the "UpgradeAtPowerCycle" -value? That's for turning the feature ON, and you wanted to turn it off? Then you should replace that with the value "manual" (also in my script in the image)

Check out the link "Managing VMware with PowerShell -- Frequently Asked Questions" http://communities.vmware.com/docs/DOC-4210

for some useful tips.

Keep on scripting! Smiley Wink

Reply
0 Kudos
henketh
Contributor
Contributor

...and yes, it's a CR after each line.

Reply
0 Kudos
gdesmo
Enthusiast
Enthusiast

Ok here is the latest failure Smiley Happy

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; [Reflection.Assem

bly]::LoadWithPartialName("vmware.vim")

GAC Version Location

--- -


-


True v2.0.50727 C:\WINDOWS\assembly\GAC_MSIL\VMware.Vim\1.0.0.0__10980...

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec = N

ew-Object VMware.Vim.VirtualMachineConfigSpec

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec.Too

ls = New-Object VMware.Vim.ToolsConfigInfo

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec.Too

ls.ToolsUpgradePolicy = "Manual"

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; Get-VM | %{(Get-V

iew $_.ID).ReconfigVM(vmConfigSpec)}

Missing ')' in method call.

At line:1 char:40

+ Get-VM | %{(Get-View $_.ID).ReconfigVM(v &lt;&lt;&lt;&lt; mConfigSpec)}

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt;

I also tried the script in the managing powershell FAQ. That looked a little different than yours. It also did not complete.

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vm =

&gt;&gt;

&gt;&gt; Get-View (Get-VM VMName).Id $vmConfigSpec = New-Object VMware.Vim.VirtualMach

ineConfigSpec

&gt;&gt;

Get-VM : 10/20/2008 10:02:52 AM Get-VM VM with name 'VMName' not foun

d, using the specified filter(s).

At line:2 char:17

+ Get-View (Get-VM &lt;&lt;&lt;&lt; VMName).Id $vmConfigSpec = New-Object VMware.Vim.Virtu

alMachineConfigSpec

Get-View : The argument cannot be null or empty.

At line:2 char:9

+ Get-View &lt;&lt;&lt;&lt; (Get-VM VMName).Id $vmConfigSpec = New-Object VMware.Vim.Virtu

alMachineConfigSpec

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec.Too

ls = New-Object VMware.Vim.ToolsConfigInfo

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt;

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt; $vmConfigSpec.Too

ls.ToolsUpgradePolicy = "Manual" $vm.ReconfigVM($vmConfigSpec)

Unexpected token 'vm' in expression or statement.

At line:1 char:54

+ $vmConfigSpec.Tools.ToolsUpgradePolicy = "Manual" $vm. &lt;&lt;&lt;&lt; ReconfigVM($vmCon

figSpec)

PS C:\Program Files\VMware\Infrastructure\VIToolkitForWindows&gt;

Reply
0 Kudos
gdesmo
Enthusiast
Enthusiast

Ok I figured it out. I was missing a dollar sign in the below line. Right before vmConfigSpec. Thanks

Get-VM | %{(Get-View $_.ID).ReconfigVM($vmConfigSpec)}

I am assuming even though this value will be changed. It will not become active until the vm is re-booted? Because the vmx is already loaded in memory. A vmotion could still initiate a tools upgrade?

Reply
0 Kudos
ploleary
Contributor
Contributor

Has anyone seen this again? Had 15 guests upgrade tools and reboot the other night after an upgrade to U4 and 158874 - DRS migrated guests back to the remediated host after it came out of maint mode. The guests were on the January tools release so they should have had the U3 patch that is referred to below. In the logs of a host it has:

tools: 05/06/09 03:42:20 Minor upgrade - migrating the tools conf settings

But then proceeds to do a full upgrade and reboot.

Anyone else seen this behaviour?

Reply
0 Kudos