Guest Customization Failure in 3.5/2.5

[Vandalay]

First VM deployment from a template on ESX 3.5i/VC2.5 for us and I understand the sysprep/customization process changed in 3.5/2.5. The template is Windows 2003 Std R2 SP2 x64. When we deploy it and use Guest Customization, the resulting VM is identical to the original template - no customization occurs. No errors are reported. On first boot, we do not see the sysprep mini-setup.

However, whenever it's shut down or boots we see a flash of a grayish screen with the words "VMware image customization is in progress...". It's only there for a second and then the system will go ahead and boot or shutdown as usual. But, no customization ever occurs.

I went ahead and manually sysprep'd one of the VMs and it went normally, but the flash of "VMware image customization is in progress..." still occurs every time on boot up and shut down.

We've done this several times now with the same results. We confirmed we have the correct sysprep files on the VC server and in the right directory in "\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\sysprep\svr2003-64".

Is this a bug? Anyone else seeing it? Should I be concerned about those VMs deployed now that flash "VMware image customization is in progress..." every time the boot?

[Vandalay]

Bump.......

Am I really the only one???

[SWCS833]

No your not vandalay. I had the same problem and no one was able to help me on here nor was VMWare Support. Everyone, even VMWare, kept blaming it on the Sysprep Version (SP2 vs SP1) and the location of the files on the VC Server. What I discovered though was that if I installed a host with Windows 2003 SP1, install just SP2 (no other updates) and then try to deploy the template, it worked. I am not sure which update beyond SP2 from windowsupdate breaks the VMWare Sysprep, but I am thinking maybe IE 7.

Let me know if this doesn't work and I can look at my template again, but I did just install an w2k3 SP1 VM and just installed SP2 and no other updates.

Also, what changed in the sysprep/customization in 3.5/2.5?

[tgradig]

I'm having similar problems. After upgrading to VC 2.5 we started having problems with our customization. It will do part of the sysprep, but not all of it. It won't add the server to our domain. It leaves it in a workgroup. If I look at the logs, c:\windows\temp\vmware-imc I get an errors in the log:

1. Incompatible service pack version of sysprep.exe. Sysprep version 1, os service pack 2.

2. The specified domain either does not exist or could not be contacted. I'm wondering if VMware changed something in the NIC settings so it isn't on the wire while trying to add it to the domain.

I didn't change the sysprep files before or after the upgrade.

Any ideas.

Tony G

[Vandalay]

Can anyone describe exactly what they changed in the way they do Guest Customization in 3.5/2.5?

[RoelVdB]

No solution here - just to say we have the same problem. x64 windows 2003.

[tgradig]

Still no resolution, but this is what Support said. Basically, they tightend the sysprep laws. Don't know why, they are killing us.

With VC 2.5, we're imposing new requirements on the version of sysprep people use with guest customization. It's described in some detail in the BSA ( Basic System Administration ) guide at http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_admin_guide.pdf page 352, please review the requirements and let me know if fixes the problem.

This is a known issue:

problem: Customization option is grayed out in the VC UI, and the text explains that windows customization resources are not installed. The sysprep/1.1 has been populated with sysprep 1.1 obtained from Microsoft. This worked OK for VC 2.0.x, but fails in 3.5.

cause: In the previous implementation of customization, we were able to be more relaxed about which sysprep was used, because we were not actually running sysprep.exe. Customization simply mounted the guest system volume and scheduled sysprep directly by making a few registry/filesystem changes. The process only used one of the sysprep files - the SID changer setupcl.exe, for which the 1.1 version worked on all the supported systems. In the new implementation, we're actually running sysprep.exe in the guest, which is very different on different systems, so we tightened up the requirements. There are now specific directories for each guest, as well as 64-bit flavors. These must be populated with the appropriate sysprep files from the Windows CD or customization will be disabled.

solution: Populate the appropriate directory in COMMON_APPDATA/VMware/Vmware VirtualCenter/sysprep with the sysprep files from the guest OS installation disk as described in Appendix B of the Basic System Administration manual.

[Vandalay]

Problem is we're using the sysprep off the same disk we installed from. The option is not grayed out, it lets us use the customization wizard, it just doesn't seem to complete the customization and somehow gets hung up - see the original post.

[tgradig]

I think there are multiple problems going on here, but I think they are all generated from the same problem. Try to clone a server across a different datacenter and see if it gets past the customization piece. In our case, it wouldn't add to the domain. Yesterday, I just cloned across two datacenters that I have setup and it worked. It added the server to the domain. If I try in the same datacenter, it's a no go.

Can you try and clone across a datacenter and see if the clone/customization completes?

Tony G.

[tgradig]

Vandalay,

Try adding the DNS Servers in the customization template of the nic and see if the server runs through the customization completly and see if the errors go away. This helped fix our problem.

Tony G.

[K-MaC]

I am also running into issues using the sysprep tools for 2k3 EE. I am able to set timezone, PC name, Windows License and specify a new SID. However any changes that I make to network configurations do not take effect. This is quite annoying as I need to create a large number of VM's with custom settings and dont want to waste my time configuring them all manually. I would assume that the domain issue is related to the networking issue however I am not sure. I suppose I could resolve the issue by adding a dhcp server but I would like to find out the reason why this doesnt work.

Sincerely

Kevin MacDonald

[tgradig]

We've never had much luck getting the IP address to take on the clones. It would always blank it out afterwards again. Our problem was resolved when we added the DNS Servers in the cloning process. Support wasn't helpfull at all resolving the problem. I got my answer by talking to my local SE.

[haen]

Yesterday and today I have checked probably 100 different threats here in the forum and various places and about 8-10 of them describe the same issue. With ESX-3.02 / VC 2.0x, the customization worked fine. Now after upgrading to 2.5/3.5 Update 1, the gues customization on Windows 2003 Server does NOT work anymore.

As I read all those threads, I can summarize that I did all the checks suggested:

- I verified multiple times and even downloaded new Sysprep files for all OSes, and put them again into the VirtualCenter Server

(there are new sysprep packages for XP SP3 and for Win2003 Server SP 2, so I used these new packages)

- I used DHCP settings as well as fixed IP

- For some time I had TEMP/TMP directories pointing to C:\TEMP, then I moved them back to %SystemRoot%/Temp

- I blanked out the Administrator password

- The Windows 2003 Template is a NEW isntallation on ESX 3.5 U1, it is NOT an old template, if was installed freshly on ESX 3.5 U1 from the original CD's (not via sysprep)

This all did not solve the problem. The Wizard runs through, either manually or using a customization template (with the same result), the Clone is created, the Clone boots and then nothing happens, no reboot. When I log in I can see that no customization was done, Hostname is still the same from the template, Administrator password is unchanged, no new SUID created... bottom line, no customization done.

When booting the system very briefly shows the "VMWare guest customization is running", but only for a second, then it returns to the normal login screen, so the sysprep is not really taking off.

I investigated further: a "vmware-inc" directory created is under

C:\Windows\Temp with two files: guestcust.log and toolsDeployPkg.log.

The toolsDeployPkg.log looks good:

-

---

1. Starting deploy pkg operation

Extracting C:\WINDOWS\TEMP\vmw1.tmp to C:\WINDOWS\TEMP\vmw2.tmp

Expanding C:\WINDOWS\TEMP\

Expanding C:\WINDOWS\TEMP\vmw2.tmp\guestcustutil.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\imgcust-reboot.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\bootrun.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\bootrunMsg.dll

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\cmdlines.txt

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\finalizationsetup.bat

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\finalize.bat

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\guestcustutil.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\imgcust-reboot.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\i386\$OEM$\loginblocker.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprep\sysprep.inf

Expanding C:\WINDOWS\TEMP\vmw2.tmp\deploy\setupcl.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\deploy\sysprep.exe

Expanding C:\WINDOWS\TEMP\vmw2.tmp\sysprepDecrypter.exe

Package command = guestcustutil.exe customize -sealparam "-quiet -mini -reseal -reboot" -nics 4000 -schedulenativeunobfusc

Running expanded command: C:\WINDOWS\TEMP\vmw2.tmp\guestcustutil.exe customize -sealparam "-quiet -mini -reseal -reboot" -nics 4000 -schedulenativeunobfusc

Command completed with exit code -1

Package deploy was successful

1. Closing log

-

---

but the other logfile contains an error at the end, about a file that cannot be found. But it does not say which file it can not find:

-

---

Customization in progress set to 1 at 2008-Jul-02 12:46:46

Rpci: Sending request='deployPkg.update.state 4 0 C:\WINDOWS\TEMP\vmware-imc\guestcust.log'

Rpci: Sent request='deployPkg.update.state 4 0 C:\WINDOWS\TEMP\vmware-imc\guestcust.log', reply='', len=0, status=1

Successfully opened key SYSTEM\CurrentControlSet\Control\Session Manager\

Size of reg_multi_sz 21.

Read multi_sz value from registry autocheck autochk *, size 21.

string value from registry autocheck autochk *.

Returning 1 elements

Got BootExecute from session mgr.

Native App sysprepDecrypter.exe, arguments ''

Copied file sysprepDecrypter.exe to C:\WINDOWS\system32\sysprepDecrypter.exe

Updated boot execute value.

Successfully created/opened key SOFTWARE\VMware, Inc.\Guest Customization\

New boot execute:

autocheck autochk *

sysprepDecrypter.exe

System Drive: C:

Deleting directory C:\sysprep if it already exists

Moving directory sysprep to C:\sysprep

select * from win32_networkadapter where Manufacturer != 'Microsoft' and manufacturer is not null and MACAddress is not null

Found 1 objects. Pointer b80318. return code 0(0x0)

Found 1 objects. Pointer b82008. return code 0(0x0)

Found 1 objects. Pointer b80740. return code 0(0x0)

Found 0 objects. Pointer 0. return code 1(0x1)

Returning value 00:50:56:84:37:E7 for system property

Returning value B2:43:20:52:41:53 for system property

Returning value 00:50:56:84:37:E7 for system property

Setting dhcp for nic # 0

Returning value
TVMESXI386W2K3\ROOT\CIMV2:Win32_NetworkAdapter.DeviceID="1" for system property

ASSOCIATORS OF {
TVMESXI386W2K3\ROOT\CIMV2:Win32_NetworkAdapter.DeviceID="1"} where ResultClass = Win32_NetworkAdapterConfiguration

Found 1 objects. Pointer b82008. return code 0(0x0)

Found 0 objects. Pointer 0. return code 1(0x1)

Clearing gateway ip addresses.

Enabling DHCP on the computer

Returning value
TVMESXI386W2K3\ROOT\CIMV2:Win32_NetworkAdapterConfiguration.Index=1 for system property

Setting DNS Server to ip <none>

Getting method object for method name SetDNSServerSearchOrder

Set status called with flag 0, result 0

Returning value
TVMESXI386W2K3\ROOT\CIMV2:Win32_NetworkAdapterConfiguration.Index=1 for system property

Setting dhcp for nic # 1

Returning value
TVMESXI386W2K3\ROOT\CIMV2:Win32_NetworkAdapter.DeviceID="9" for system property

ASSOCIATORS OF {
TVMESXI386W2K3\ROOT\CIMV2:Win32_NetworkAdapter.DeviceID="9"} where ResultClass = Win32_NetworkAdapterConfiguration

Found 1 objects. Pointer b80318. return code 0(0x0)

Found 0 objects. Pointer 0. return code 1(0x1)

Clearing gateway ip addresses.

Rpci: Sending request='deployPkg.update.state 4 101 C:\WINDOWS\TEMP\vmware-imc\guestcust.log@The system cannot find the file specified.

'

Rpci: Sent request='deployPkg.update.state 4 101 C:\WINDOWS\TEMP\vmware-imc\guestcust.log@The system cannot find the file specified.

', reply='', len=0, status=1

error number 0, The system cannot find the file specified.

-

---

Does anyone know where this problem comes from and how it can be solved?

Kind Regards, Herbert

[K-MaC]

Hello Herbert. I am now running into the exact same issue as you. I have tried with multiple OS's as well. I am really at a loss as to why its not working. I see the VMware sysprep splash screen numerous times but nothing actually gets updated. I've tried all of the same things as you have to no avail. It's quite disheartening seeing as how I have to build a tonne of clients next week and would really really rather not do it manually. Is there no one who has a resolution for this?

Sincerely

Kevin MacDonald

[haen]

Another solutions tip, which resolved my cloning/sysprep problem on Windows 2003 Server:

Symantec Endpoint Security (SEP)... I had previously read a tip to disable any Antivirus-Services. I had tried that, but obviously I had not disabled it completely only partially

This specific software, SEP, which is per our company policy REQUIRED for all windows machines and VM's (therefor it is already part of all templates!) includes not only an Antivirus module, but also a "threat protection" module, which is supposed to work against network or local threats based on "software behaviour". So obviously this software recognized the VMWare/sysprep integration and thinks it is a "virus" or "trojan" and prevented it from running.

After I disabled the threat protection part of SEP and only installed the Antivirus component, the customization process works without problem. This is the solution for Windows-Server (here only Antivirus is required, the other modules optional), but not for XP, where the threat protection components is also required.

For me the cause is clear now, hope this helps others.

VMWare: I hope someone is reading this. I would suggest you get in touch with Symantec and other vendors and explain this topic. I am pretty sure a "rule" can be included in threat protection components to avoid interfering with the Sysprep Customization of VMWare Images. I guess that many companies require the use of such software, so this is a significant problem.

Kind Regards, Herbert

[K-MaC]

Hello Herbert, thanks for your tip. It seems my trouble was the same as yours. Symantec SEP causes the VMware Configuration to not work correctly. I found that I had to completely remove the program to get it to work. I tried to disable SEP and ran into the same problem.

Cheers

Kevin MacDonald

[Vandalay]

Promising lead, I think my problem may have started about the time I added McAfee antivirus to the template. I will test and see if this fixes it for me.

Thanks.

[meistermn]

Could be that the anti virus client should not be in the gold image like shown on page 3 of esx2 best pratice

From design it should be better only have to have the os and the vmware nic driver , not the complete vmware tools, in the gold image and create a folder for the software like antivirus that should be installed after cloning , syspreping and joining the domain.

So after cloning the vm and after sysprep did their job and join the domain, after the first login script the antivirus could be installed from local disk

per run, runonce registry or through autostart. This would keep the gold image easy to update. because you only have to modify the software folder.

The disadvantage is that the install time takes longer. If you what shorter install times, than install more software in the image for e.g. .net , powershell

and so on.

So the design of the vm plays the key roll.

[K-MaC]

Hey Meistermn, thanks for the advice. I agree with your post that the anti-virus client should definately be removed from the gold image. Perhaps if someone from VMware is still reading this thread they could look into the issue. However, for the most I've not seen any posts on the subject of Customization Specifications from any of the VMware staff on this site. In all honesty I wasted a fair amount of time trying to get to the bottom of this problem, which could have been solved by adding one line of text to any number of pdf guides available. One can only hope that this information is added for the benefit of others.

Cheers

Kevin

[urbachr]

Hello,

I am having this same problem of the customization wizard not running fully. I also get the sysprep inompatible message in the vm log file. Is there any resolution to this problem? We are starting a large deploy of VM's and I would like to use the wizard and trust that it is doing what it should be doing.

Thanks,

Ryan