VMware Image Customization after every power cycle

[dvlowe]

Here is one I can not find any info on.  I created a clean Windows 2012 R2 template and customization.  When I deploy the template the customization works and the servers gets configured correctly.  Every time I fully power off the VM then power it back on the server will run the VMware Image Customization after a couple minutes.  It does not run the VMware Image Customization after reboots, only through power cycles.  Anyone seen this or have some insight on how I can stop this from happening?

We are on vCenter 5.1 Update 2 (we cannot update to 5.5 at this time).

[DigitalKiller]

We just started seeing this as well, also on 5.1 Update 2. There is this KB: VMware KB: Virtual machines repeatedly display the message: VMware Image Customization in progress that describes the issue as being:

"If a virtual machine template is created with an incorrect operating system type, future virtual machine deployments that include customization options may fail to deploy sysprep, causing the VMware image customization message to display each time the deployed virtual machine is started or if there is an interruption in the sysprep process that may result in a broken customization routine, causing sysprep to repeatedly attempt to run at boot up."

I am not sure if ours was created with the wrong OS and then changed at a later date, but when I checked, it was correct. So far we we have tested 2008 R2 Standard an 2008 R2 Enterprise and have seen the issue with multiple customization specs. Unfortunately the KB explains how to fix the VM, but not how to keep it from happening in the first place. I think I may open a ticket with support tomorrow.

[dallasdrake]

This is a bug in the latest build of 5.1U2 as well as 5.5. There is currently no fix, however there is a workaround.

The customization logfile: C:\windows\temp\vmware-imc\guestcust.log contains the following error:

[2013-05-15T16:24:48 ERROR] Unable to set customization status in vmx

Workaround:

1. Shutdown a vm.

2. Connect to the host using ssh or console

3. CD to the VM folder.

4. Remove the following like from virtual machines .vmx file after the initial customization of the Guest OS has completed.

     tools.deployPkg.filename="imcXXXX.tml"

You could also power off a VM, download the .vmx file, edit it and upload it to the datastore.

I am being told that there will be a permanent fix in the next patch.

[DigitalKiller]

I opened a ticket on this finally yesterday, and was told the same thing. Apparently VMware has only known about this for ~1 week. Interestingly, I was told that currently the only affected hosts are ESXi 5.1 build 1743533, older version or newer version of this build number 1743533 are not affected. I am upgrading some hosts to 5.5 and will be testing that soon.

[ssaviers]

I usually resolve this by editing the following key on our windows 2008 R2 servers. Not sure if it applies to your situation but thought I would throw it out there.

     HKLM\SYSTEM\CurrentControlSet\Contorl\Session Manager

    Under BootExecute remove the sysprep value if present and then reboot.

[dvlowe]

Since I wasn't getting a response to my post for about a week I did end up calling support last week and the tech hadn't seen the issue and neither did his associates.  We narrowed it down to the inability to remove the line in the VMX file that launches the customization.  We confirmed this by manually removing that line.  A couple days later he confirmed that other customers were starting to call in and it had been confirmed there was a bug in one of the patches that is causing this.  He gave me the same workaround and said the team was working on the fix.  He then sent me what may be an easier workaround:

Alternate Workaround:

1. Shutdown the affected vm.

2. Edit the vm settings

3. Under Options->Advanced->General click the "Configurations Parameter..." button

4. Remove the "imcXXXX.tml" value from the line "tools.deployPkg.filename"

5. Save and test it.

Both of these workarounds accomplish the same thing (although the one dallasdrake mentioned may be preferred since you can remove the whole line) and it is just a matter of preference.

[dvlowe]

ssaviers,

This isn't the same issue you are referring which is resolved with the "usual resolution".  After the customization the everything in the registry is correct and normal restarts work flawlessly.  It is only when the VM is shutdown then starts back up that it reruns the script.

[ssaviers]

gotcha, thanks for clarifying.

[YucongSun]

I found out this is caused by

installing ESX510-20140401-BG, the solution is to not install this patch.  (or install the previous esx-base , so effectively a rollback)

In the patch notes they claim to fix the exact issue, but turns out this patch is the one actually causing it.:

VMware KB: VMware ESXi 5.1, Patch ESXi510-201404401-BG: Updates esx-base

PR1169229: During guest operating system customization, if a virtual machine is migrated because of DRS, the customization process might fail and you might notice an error message similar to the following in the Guestcust.log file:

Unable to set customization status in vmx

If someone can pass this to vmware team, it would be great.

[oendelm]

Someone knows if this issue occurs only with new virtual machines? The issue may occurs with old virtual machines (created before the upgrade to build 1743533)? i'm afraid of fully power off my old servers then power it back on and the servers start to run the VMware Image Customization.

Thanks.

[Konstantin_K]

  (or install the previous esx-base , so effectively a rollback)

But how it can be done?

I have tried to do it. But only message

~ # esxcli software vib install -d "/vmfs/volumes/EMC-VNX01-AT/ESXi510-201402001.zip" -f

Installation Result

   Message: Host is not changed.

   Reboot Required: false

   VIBs Installed:

   VIBs Removed:

   VIBs Skipped: VMware_bootbank_ata-pata-am.......

or

~ # esxcli software vib install -v "/vmfs/volumes/EMC-VNX01-AT/VMware_bootbank_esx-base_5.1.0-2.26.1612806.vib" -f

Installation Result

   Message: Host is not changed.

   Reboot Required: false

   VIBs Installed:

   VIBs Removed:

   VIBs Skipped: VMware_bootbank_esx-base_5.1.0-2.26.1612806

was recieved

[Konstantin_K]

Someone knows if this issue occurs only with new virtual machines?

Only new VMs

[YucongSun]

You need to install the esx-base from the previous patch release,

that is download from https://download2.vmware.com/patch/software/VUM/OFFLINE/release-420-20140226-922881/ESXi510-20140200...

if the link doesn't work , download this way:

1) go to https://www.vmware.com/patchmgr/findPatch.portal

2) search for esxi, 5.1

3) use the download link under ESXi510-201402001   to download the previous release

and unzip, install the esx-base vib from that.

[YucongSun]

also, you should make sure host is in "maintenance mode"

[Konstantin_K]

I tried to do same way

~ # esxcli software vib install -v "/vmfs/volumes/EMC-VNX01-AT/VMware_bootbank_esx-base_5.1.0-2.26.1612806.vib" -f

Installation Result

   Message: Host is not changed.

   Reboot Required: false

   VIBs Installed:

   VIBs Removed:

   VIBs Skipped: VMware_bootbank_esx-base_5.1.0-2.26.1612806

[YucongSun]

are your host in maintenance mode?

[Konstantin_K]

Yes, it is.

[YucongSun]

then I don't know, that's exactly what I did and it worked for me. I don't use auto-deploy either.     

You can try  pass --no-live-install   "--force" or --no-sig-check for esxcli software command, to see if it make a difference.

[Konstantin_K]

It works!

First time when I executed installation command (in PowerCli) I get some error about "operation timed out". And after that I tried to execute it in SSH, but only message "host is not changed" saw.

Really host had downgraded just first time and after reboot I can see it.

Thanks a lot!

[oendelm]

Do you know if the latest version of 5.5 has the same issue?