PatrickDLong
Enthusiast
Enthusiast

productLocker location getting changed when applying any patches via VUM?

Jump to solution

I have a fleet of hosts running ESXi 7.0 U3c - these hosts all have motherboard-based micro-SD card boot devices.  As part of the recommended mitigations to reduce I/O to this type of boot device configuration, I have from the beginning (even back when these hosts were ESXi 6.5/6.7) always configured productLocker to point to a shared storage location.  In addition to the I/O reduction to the boot device, this is a convenient way to keep all vm's tools installation current and consistent regardless of what host they are running on, because vmTools are backward and forward compatible to host version.  When a new vmtools version is released, I simply replace the files in my shared storage location (and restart mgmt agents on hosts) to compare guests' Tools version against the newly-uploaded version.

The desired productLocker symlink is shown from the ls -n output:

productLocker -> /vmfs/volumes/5e2b25a0-281972a8-507b-5cb901ffba10/SharedLocker

After upgrading these hosts from 7.0U2d to 7.0 U3c (using the vendor-custom image profile) whenever I run any subsequent VUM updates to patch drivers, etc and reboot, the productLocker value reverts to:

productLocker -> /SharedLocker

where the SharedLocker text is in red indicating it is an invalid or accessible location.  I can reset this using the API call like:

$esxName = '<hostname>.<domain>.net'
$dsName = '<Datastore label>'
$dsFolder = 'SharedLocker'
$esx = Get-VMHost -Name $esxName
$ds = Get-Datastore -Name $dsName
$oldLocation = $esx.ExtensionData.QueryProductLockerLocation()
$location = "/$($ds.ExtensionData.Info.Url.TrimStart('ds:/'))$dsFolder"
$esx.ExtensionData.UpdateProductLockerLocation($location)
Write-Host "Tools repository moved from"
Write-Host $oldLocation
Write-Host "to"
Write-Host $location

which resets the value to the desired location appropriately, and this setting DOES then persist across reboots from that point on...but if I then subsequently apply ANY additional driver update using VUM it resets the productLocker symlink value back to /SharedLocker on reboot.  Has anyone else seen this behavior or know the reason why this happens and how to prevent it?  Note, I do NOT see this behavior on my few hosts that have local high-endurance bootable media. and yes, I will be retrofitting all my hosts with M.2 disks, but I'd prefer to not have to deal with this in the interim every time I patch a host.

1 Solution

Accepted Solutions
PatrickDLong
Enthusiast
Enthusiast

No sooner do I post the above than I see that 7.0 U3f is now released... and it includes this fix!!!  I will be patching a few hosts today and will report results.

https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3f-release-notes.html 

  • PR 2964856: Path to the /productLocker directory on a datastore truncates after patching ESXi 7.x

    After the first upgrade or update of your system to ESXi 7.x, with each consecutive update, also called a patch, you might see a truncation in the path to the /productLocker directory on a datastore. For example, if your first patch on ESXi 7.x is from 7.0 Update 2 to 7.0 Update 3, the path to the /productLocker directory originally is similar to /vmfs/volumes/xxx/VMware-Tools/productLocker/. However, for each consecutive patch, for example from 7.0 Update 3 to 7.0 Update 3c, the path becomes similar to /VMware-Tools/productLocker/.

    This issue is resolved in this release.

View solution in original post

21 Replies
UKE
Contributor
Contributor

ive seen this in our environment last week after upgrading ESXi with Fujitsu ISO 7.0.3c. After rebooting all hosts showing an Host Profile Missmatch with the ProductLocker location.  

Connected via ssh, the symlink /ProductLocker got removed!

It could be replaced my applying the Host Profile again, but strange behavior, and i think vmware esxi or vum has just another/new bug 😕

 

 

 

 

stanm88
Contributor
Contributor

We are also having the same issue.  I have had several support cases open, but the issue still persists when applying updates with a host reboot.

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

I have opened a case for this as well.  Will post updates here when I have one.  I continue to see this issue when applying 7.0 U3d build 19482537 via VUM/vLCM to hosts that were previously on version 7.0 U3c

0 Kudos
stanm88
Contributor
Contributor

Here are the steps from the last ticket I opened:
1- SSH to the host
2- rm /productLocker -rf
3- ln -s <vmtoolpath> /productLocker
4- esxcli system settings advanced set -o /UserVars/ProductLockerLocation -s <vmtoolpath>
5- /sbin/auto-backup.sh (to keep settings after reboot)

This did work across reboots, but as soon as any updates are run, the symlink for productLocker no longer works.

PatrickDLong
Enthusiast
Enthusiast

Quick case update - GSS advised me to ensure that UserVars.ToolsRamdisk value was set to 1 and reboot  prior to application of VUM updates, however I proved that setting does NOT eliminate the productLocker location getting reset from  productLocker -> /vmfs/volumes/<volume_name>/SharedLocker to new value productLocker -> /SharedLocker on the reboot following any VUM application of updates.  Tried it with UserVars.ToolsRamdisk set to both 0 and 1, it does not make a difference in the result.  Uploaded more logs, still in progress.

0 Kudos
stanm88
Contributor
Contributor

I have a session set up today with VMWare support. I will post any updates that I receive.

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

Today's update:

We are..."engaging the engineering team to provide a fix for this issue"

0 Kudos
stanm88
Contributor
Contributor

My support session ended with the tech telling me to this commands in the following order:

1- SSH to the host
2- rm /productLocker -rf
3- esxcli system settings advanced set -o /UserVars/ProductLockerLocation -s <vmtoolpath>

4- ln -s <vmtoolpath> /productLocker
5- /sbin/auto-backup.sh (to keep settings after reboot)

I haven't gotten a chance to verify if the value holds after updates as all out hosts are up to date.  I did see there were some updates today, but I'll have to wait until our next patch cycle to test

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

thanks @stanm88 . But I think that's just the standard set of commands to delete and recreate the productLocker symlink to your target location? I've been doing the autobackup.sh for some time as well.   While your new productLocker symlink setting will be retained across multiple reboots so long as you do not use VUM/vLCM, I believe 100% this symlink is going to get reverted to productLocker -> /SharedLocker when you next apply any update via VUM (and perform the subsequent reboot.) Does not matter whether it is a single driver update that is performed or a minor revision upgrade like 7.0 U3c to 7.0 U3d, the symlink gets reverted by invoking VUM to do the work.  The same driver update or ESXi update applied manually via CLI followed by manual reboot does NOT revert the symlink.

0 Kudos
stanm88
Contributor
Contributor

You were correct, tested on a host this morning and the productLocker location was again reset......

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

Latest update from GSS:  "engaged our engineering team who have acknowledged this issue and are actively working on a fix. Currently in an internal testing phase"

0 Kudos
JoergenKaare
Contributor
Contributor

Just upgraded our Cisco usc blades to 7.0.3d - encounter same problem.

0 Kudos
stanm88
Contributor
Contributor

Here is what I just got back from support:

Hello (name removed),
I hope you are doing great.

We were able to reproduce the issue in our labs and it is a known bug.
This issue will be fixed in the next major release (8.0) and our back engineers are working on it.

In the meanwhile kindly go with the workaround until the next release.

Please let me know if you'd require any further assistance, I'd be more than happy to help out.
Otherwise, I'll proceed with placing this Support Request in temporary archival mode, but I'll be notified in case you decide to share any updates/inquiries with me.
 
Stay safe and have a wonderful day.
--
(name removed)
Technical Support Engineer
Global Support Services, VMware Inc.
Office Hours: Monday-Friday [8:00 AM - 4:00 PM EST]

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

My response from support included:

Engineering has confirmed that the fix will be part of next ESXi 7 u3 release, planned tentatively to be released in later part of June 2022.

(Note: Release timeline is tentative and could vary depending on internal scenarios).

 

So it may be released in a future rev of 7.0 U3 or future 8.0 - but in either case it appears to be an acknowledged bug that will have a fix.

0 Kudos
Zahni
Contributor
Contributor

Same issue here. The Problem: after installing updates, we're unable to deploy any tools update, because the Update Manager makes a wrong symlink to /ProductLocker. It links to itself. 

Fix it ASAP!!!

0 Kudos
stanm88
Contributor
Contributor

I've been running this as a powershell script for each of my servers and it seems to "fix" the location until the next update.  Hope maybe this helps someone:

connect-viserver (server IP) -User root -Password (root password) -wa 0
Get-VMHost | Get-AdvancedSetting -Name "UserVars.ProductLockerLocation" | Select-Object Entity,Value
Get-VMhost -Name (server IP) | %{$_.ExtensionData.UpdateProductLockerLocation_Task("location of your vmware tools")}

technikus
Contributor
Contributor

Hi,

same problem and what adds to it

Changing the product locker location via MOB doesn't work either. (UpdateProductLockerLocation_Task)

 

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

This fix did not make it into ESXi 7.0 U3e - or at least I should say I am still experiencing this issue when upgrading 7.0 U3d to ESXi 7.0 U3e via VUM.  Perhaps Support gave me incorrect information, or perhaps this fix is in there, but only protects ProductLocker from reverting on updates applied AFTER installing 7.0 U3e.  I'll report back as soon as I apply any post-U3e patches.

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

No sooner do I post the above than I see that 7.0 U3f is now released... and it includes this fix!!!  I will be patching a few hosts today and will report results.

https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3f-release-notes.html 

  • PR 2964856: Path to the /productLocker directory on a datastore truncates after patching ESXi 7.x

    After the first upgrade or update of your system to ESXi 7.x, with each consecutive update, also called a patch, you might see a truncation in the path to the /productLocker directory on a datastore. For example, if your first patch on ESXi 7.x is from 7.0 Update 2 to 7.0 Update 3, the path to the /productLocker directory originally is similar to /vmfs/volumes/xxx/VMware-Tools/productLocker/. However, for each consecutive patch, for example from 7.0 Update 3 to 7.0 Update 3c, the path becomes similar to /VMware-Tools/productLocker/.

    This issue is resolved in this release.