VMware Cloud Community
glloydclose
Contributor
Contributor

vShield Endpoint Upgrade from v3 to v5 vfile issues

I am in the middle of an upgrade of our clusters from vSphere 4.1 to vSphere 5.

On all vSphere 4.1 hosts we had vShield Endpoint 3.0.8-308978 installed (epsec_vfile).

On the first cluster, to perform the upgrade, I left all VMs on a single vSphere 4.1 host and rebuilt the remaining hosts manually via ISO to ESXi v5 and applied a patch to upgrade ESXi to v5.0.0 build 474610.

I then installed the vShield Manager v5 appliance and deployed vShield Endpoint v5 to all the vSphere 5 hosts.

My issue is I cannot vmotion or power on any of the VMs from the remaining vSphere 4 host to the vSphere 5 hosts. Because we had installed the epsec_vfile filter (Endpoint v3) on all hosts, all VMs have had their .vmx file updated, which is expected. What isn't expected is that I cannot power on these VMs on vSphere 5 hosts with Vshield Endpoint v5 installed. I get the following errors:

Power On virtual machine:Module DevicePowerOn power on failed.

An unexpected error was received from the ESX host while powering on VM vm-266.

Module DevicePowerOn power on failed.

Unable to create virtual SCSI device for scsi0:0, '/vmfs/volumes/4efb7a66-5903dc3b-22b1-0025b5010116/PDC1ADG002/PDC1ADG002.vmdk'

Failed to attach filter 'VFILE' to scsi0:0: Not found (195887107).

As per KB1030463, If I remove the following two lines from the .vmx file, the VMs power on fine:

VFILE.globaloptions = "svmip=169.254.50.39 svmport=8888"
scsi0:0.filters = "VFILE"

This is also expected, but should not be necessary. This is the recommended workaround if you want to power on a VM after it has been powered on, on an ESXi host with vShield Endpoint v3 installed.

On vShield Endpoint v3 the .vmx file is updated on power on, if the above lines are not present. On vShield Endpoint v5 it does not appear to be updating the .vmx files of the VMs after I have removed them.

I have logged into the hosts via SSH and confirmed the epsec filter is installed:

    2012-01-28T01:39:27.128917+00:00: The following VIBs are

      installed:

        epsec-mux     5.0.0-447150

Is it just me does it appear vShield Endpoint v3 and Vshield Endpoint v5 are not compatible?
Reply
0 Kudos
7 Replies
glloydclose
Contributor
Contributor

To answer my own question...

vShield Endpoint v3 and vShiend Endpoint v5 are NOT compatible.

vShield endpoint v3 loads a kernel module called vfile on vSphere 4.1 host boot up which updates the .vmx files of any VMs running on the host.

vShield endpoint v5 does not! No vfile kernel module is loaded, no vfile line entries are added to the .vmx files of the VMs.

This is implied in the vShield 5 Quick Start guide on Page 31 under ‘Upgrading vShield Endpoint’:

2. Deactivate all Trend DSVAs. This is required to remove vShield related VFILE filter entries from the virtual machines.

I missed that step unfortunately. It meant I couldn’t power on any of my VMs on the vSphere 5 hosts and had to manully update the .vmx file of every virtual machine to remove the vfile line entries as per KB1030463.

My advice to anyone upgrading from vShield 4.1 with vShield Endpoint v3 and Trend DS 7.5 is to make sure they do a proper decommissioning exercise, even if they are doing a clean install and not an upgrade:

  1. Remove vShield Endpoint fitler from all hosts (epsec_vfile)

  2. Remove Trend filter from all hosts (dvfilter-dsa)

  3. Deactivate all Trend Appliances

I would still recommend a clean install and not an upgrade as the cleaner approach to performing an upgrade from vSphere 4.1 to vSphere 5.0.

When you are deploying Trend 8, don't forget to resynchronise, or even better remove the old vCenter from Trend Manager 8 and add it back to update your Virtual Center and vShield settings.

This will ensure your Trend DS 8 appliances are able to activate and register as security VMs on your new vSphere 5 hosts.

Reply
0 Kudos
iforbes
Hot Shot
Hot Shot

Hi glloydclose. I'm about to go though this excercise. My customer has Trend DS 7.5 protecting their vm's. I'll be working with them to upgrade their ESX servers from 4.1 to 5.0.1, and also upgrading vShield manager to version 5, so that we can upgrade Trend DS to version 8.

Now, what's different is that my client will be performing a clean install of ESXi 5 onto their ESX 4 servers because of a RAID misconfiguration. Is your suggestion for them to deactivate the DSVA's on the ESX 4 servers first, so that the VFILE entries in the vm's will all get removed? That would seem the way to go.

I don't think we need to worry about the first two steps that you mention because we will be completely wiping the ESX 4 servers and installing ESXi 5. We will then have ESXi5 servers, vShield 5 and clean vm's (no VFILE entries in the .vmx). At this point I think I will install Trend DS 8.0 as a clean install - just seems cleaner than an upgrade.

Thoughts?

Thanks

Reply
0 Kudos
glloydclose
Contributor
Contributor

Hi Ian,

Your thinking is correct. Even though your client is rebuilding their ESXi hosts from scratch, they will need to deactive Trend protection of all their VMs, before they blow away the host's virtual appliance. Remember if you vmotion VMs that have been deactivated, onto a host that has an active DS 7.5 virtual appliance, the .vmx files will get re-written so probably best to turn off DRS too.

I documented the right way to do things on my blog, step by step: http://blocksandbytes.com/2012/02/01/vsphere-5-vshield-5-trend-ds-8-vblock-300hx-upgrade/

Another key thing to remember is to also remove any imported virtual center in Trend Deep Security Manager to clear out the old config for virtual center and vshield. Each virtual center object imported into the DSM also has the vShield Manager credentials so its easier just to blow it away and add virtual center back in after you've finished your upgrade work.

If they are worried about losing their existing Trend VM config, they can update the vshield credentials after the upgrade in the DSM if required and re-synchronise with vcenter to force through all the changes that have been made to the ESXi hosts, appliances, etc. This might need to be done regularly as you'll be making so many changes.

Another thing to remember is that the vShield 4.1 Endpoint driver for Windows in vSphere 4 is a standalone install and will need to be uninstalled before you can install vShield 5 Endpoint Driver which is included in VMware Tools in Vsphere 5. Unfortunately the vShield driver is not part of the default components so you cannot use an automated install. Either go interactive and select the vshield endpoint component or use the command line below, which will do a silent install with the right config.

Take out the Remove="Hgfs,ThinPrint" if you want Shared Folders and ThinPrint component installed. I remove Shared Folders as it just seems to cause issues, especially on Domain Controllers and we don't use Thinprint.

msiexec.exe /i "vmware tools64.msi" ADDLOCAL=ALL REMOVE="Hgfs,ThinPrint" /qn /l*v c:\vmwaretools64-install.log /norestart

good luck.

Reply
0 Kudos
iforbes
Hot Shot
Hot Shot

This is great. Thanks for the information. I forgot about the Endpoint thin agent, so thanks for the reminder about that. Nice that it's included in VMware Tools for ESXi 5. We will be doing a VMware hardware and tools upgrade for all the vm's after we upgrade ESX from 4 to 5, but you suggest we still have to uninstall the 4.1 endpoint thin driver first? Will it cause issues if we just install the new VMware Tools which includes the new Endpoint thin driver (version 5)?

Is there a way to automate the removal of the Endpoint thin agent version 4.1 from all vm's, or does it have to be done interactively for each vm?

Reply
0 Kudos
JonathanG
Enthusiast
Enthusiast

How about:

msiexec /x filename.msi /q

For example: msiexec /x VMware-vShield-Endpoint-Driver-1.0.0-402356.x86_64.msi /q

Then use any standard Windows tools to execute that command on each VM, for example

psexec \\Server -u "DOMAIN\Username" -p "PASSWORD" cmd /c "msiexec.exe /i "VMware-vShield-Endpoint-Driver-1.0.0-402356.x86_64.msi" /uninstall /quiet /norestart"
Reply
0 Kudos
glloydclose
Contributor
Contributor

Hi Ian,

Yes, unfortunately you have to uninstall the old version.


If you try install the new version with the old version still installed it just errors out. Poor programming on VMware's part. They could have made the installer a bit more intelligent.


It really screwed up my whole automated vmware tools deployment via VUM, because I couldn't figure out how to customise a VUM install of VMware Tools so the vShield Endpoint Driver never installed. Real pain in the arse.

Normally I would use the syntax "msiexec /x {ProductCode} /qn /l*v c:\vmwaretools64-uninstall.log /norestart"


The ProductCode for vShield Endpoint 1 on my machine is {377C5447-085C-40DB-BEB2-6C242BA4B4DE} so

msiexec /x {377C5447-085C-40DB-BEB2-6C242BA4B4DE} /qn /l*v c:\vmwaretools64-uninstall.log /norestart

should do the trick. you can verify the productcode here -- HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\

reboot before you install VMware Tools though.

Gareth

Reply
0 Kudos
iforbes
Hot Shot
Hot Shot

Thanks for this information. Great stuff.

Reply
0 Kudos