cesprov's Posts

Seeing a strange issue with VMware Tools here that I don't see anyone else talking about in respect to the Spectre variant 2 vulnerability.  I have a vSphere environment that is fully patched through the 6.0U3e vCenter and the ESXi 3/20/18 patches.  Within that environment, I have a bunch of CentOS 6.9 servers.  Now we have slacked off for quite some time on upgrading the VMware Tools within these CentOS VMs so most are still running a 5.1.x version, 9.10.1.47876 (build-2791197).  These VMs are fully patched with the latest kernel so they shouldn't be vulnerable to Meltdown or Spectre.  Prior to upgrading the VMware Tools, the Meltdown/Spectre check script, found here, reports it has not being vulnerable to all three, as shown below: CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' * Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active) * Kernel has array_index_mask_nospec (x86):  NO * Kernel has the Red Hat/Ubuntu patch:  YES * Kernel has mask_nospec64 (arm):  NO > STATUS:  NOT VULNERABLE  (Mitigation: Load fences) CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' * Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active) * Mitigation 1   * Kernel is compiled with IBRS/IBPB support:  YES   * Currently enabled features     * IBRS enabled for Kernel space:  NO     * IBRS enabled for User space:  NO     * IBPB enabled:  NO * Mitigation 2   * Kernel has branch predictor hardening (arm):  NO   * Kernel compiled with retpoline option:  YES   * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation) > STATUS:  NOT VULNERABLE  (Mitigation: Full retpoline) CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' * Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active) * Kernel supports Page Table Isolation (PTI):  YES  (found 'CONFIG_PAGE_TABLE_ISOLATION=y') * PTI enabled and active:  YES * Running as a Xen PV DomU:  NO > STATUS:  NOT VULNERABLE  (Mitigation: PTI) Then if I upgrade to the latest version of VMware Tools for 6.0U3, 10.1.10.63510 (build-6082533), and re-run the script, it now shows the VM is vulnerable to Spectre variant 2: CVE-2017-5753 [bounds check bypass] aka 'Spectre Variant 1' * Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active) * Kernel has array_index_mask_nospec (x86):  NO * Kernel has the Red Hat/Ubuntu patch:  YES * Kernel has mask_nospec64 (arm):  NO > STATUS:  NOT VULNERABLE  (Mitigation: Load fences) CVE-2017-5715 [branch target injection] aka 'Spectre Variant 2' * Mitigated according to the /sys interface:  NO  (kernel confirms your system is vulnerable) * Mitigation 1   * Kernel is compiled with IBRS/IBPB support:  YES   * Currently enabled features     * IBRS enabled for Kernel space:  NO     * IBRS enabled for User space:  NO     * IBPB enabled:  NO * Mitigation 2   * Kernel has branch predictor hardening (arm):  NO   * Kernel compiled with retpoline option:  YES   * Kernel compiled with a retpoline-aware compiler:  UNKNOWN > STATUS:  VULNERABLE  (Vulnerable: Retpoline with unsafe module(s)) CVE-2017-5754 [rogue data cache load] aka 'Meltdown' aka 'Variant 3' * Mitigated according to the /sys interface:  YES  (kernel confirms that the mitigation is active) * Kernel supports Page Table Isolation (PTI):  YES  (found 'CONFIG_PAGE_TABLE_ISOLATION=y') * PTI enabled and active:  YES * Running as a Xen PV DomU:  NO > STATUS:  NOT VULNERABLE  (Mitigation: PTI) Using this to determine what the "unsafe module(s)" are shows: VULNERABLE - No Retpoline found - vsock VULNERABLE - No Retpoline found - vmci Obviously these are VMware Tools components.  These were not reported as a problem with VMware Tools 9.10.1.47876 (build-2791197) but are being reported with 10.1.10.63510 (build-6082533).  What's the deal?  Are these VMs that are reporting those two components really vulnerable?  And if so, when can we expect VMware to fix this?

Been trying to upgrade to 6.0U3e (from 6.0U3d) all morning and it is constantly failing on the "VMware python components" right at the beginning of the install with the dreaded 3010 error.  I've upgraded this vCenter countless times so I am familiar with all the files that get locked and cause 3010s (see VMware Knowledge Base).  This is the first time I have seen the 3010 appear on the Python components part though.  Prior to the install, none of the python files from C:\Program Files\VMware\vCenter Server\python are open in memory, as seen through Process Explorer.  While the "VMware Python components" section is displaying in the upgrade, if you refresh a search for open "python" files in Process Explorer, you can clearly see that the install itself has now opened quite a few files in C:\Program Files\VMware\vCenter Server\python.  The install then crashes with a 3010 error.  Checking the pkgmgr-comp-msi.log afterwards, you see these errors occurred: MSI (s) (A8:48) [12:49:37:515]: Executing op: RegisterSharedComponentProvider(,,File=python27.dll,Component={8C4A7097-E243-54DF-82C4-F45CA52DE496},ComponentVersion=2.7.13150.1013,ProductCode={DDB1C8B6-82DB-4FD5-A7AB-B1DA7B31E771},ProductVersion=6.0.0,PatchSize=0,PatchAttributes=0,PatchSequence=0,SharedComponent=0,IsFullFile=0) MSI (s) (A8:48) [12:49:37:531]: Executing op: FileCopy(SourceName=python27.dll,SourceCabKey=python27.dll,DestName=python27.dll,Attributes=1536,FileSize=3046328,PerTick=65536,,VerifyMedia=1,,,,,CheckCRC=0,Version=2.7.13150.1013,Language=0,InstallMode=126353408,,,,,,,) MSI (s) (A8:48) [12:49:37:547]: File: C:\Program Files\VMware\vCenter Server\python\python27.dll; Overwrite; Won't patch; Existing file is of an equal version MSI (s) (A8:48) [12:49:37:547]: Source for file 'python27.dll' is compressed MSI (s) (A8:48) [12:49:37:547]: Re-applying security from existing file. MSI (s) (A8:48) [12:49:37:547]: Note: 1: 2205 2:  3: Error MSI (s) (A8:48) [12:49:37:547]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1603 MSI (s) (A8:48) [12:49:37:547]: Product: VMware-python. The file C:\Program Files\VMware\vCenter Server\python\python27.dll is being used by the following process: Name: python , Id 5368. MSI (s) (A8:48) [12:49:37:547]: Verifying accessibility of file: python27.dll Info 1603.The file C:\Program Files\VMware\vCenter Server\python\python27.dll is being held in use.  Close that application and retry. MSI (s) (A8:48) [12:49:36:938]: Executing op: FileCopy(SourceName=python.exe,SourceCabKey=python.exe,DestName=python.exe,Attributes=1536,FileSize=42936,PerTick=65536,,VerifyMedia=1,,,,,CheckCRC=0,,,InstallMode=126353408,HashOptions=0,HashPart1=269396038,HashPart2=-283905172,HashPart3=994898176,HashPart4=-1453880127,,) MSI (s) (A8:48) [12:49:36:938]: File: C:\Program Files\VMware\vCenter Server\python\python.exe; Overwrite; Won't patch; Existing file is unversioned and unmodified - hash doesn't match source file MSI (s) (A8:48) [12:49:36:938]: Source for file 'python.exe' is compressed MSI (s) (A8:48) [12:49:36:938]: Re-applying security from existing file. MSI (s) (A8:48) [12:49:36:938]: Note: 1: 2205 2:  3: Error MSI (s) (A8:48) [12:49:36:938]: Note: 1: 2228 2:  3: Error 4: SELECT `Message` FROM `Error` WHERE `Error` = 1603 MSI (s) (A8:48) [12:49:37:375]: Product: VMware-python. The file C:\Program Files\VMware\vCenter Server\python\python.exe is being used by the following process: Name: python , Id 5368. MSI (s) (A8:48) [12:49:37:375]: Verifying accessibility of file: python.exe Info 1603.The file C:\Program Files\VMware\vCenter Server\python\python.exe is being held in use. Close that application and retry. It's pretty clear that the installation is creating its own locks on the files in C:\Program Files\VMware\vCenter Server\python and thus causing the 3010 errors.  The only way I was able to get around this problem was to launch the install and before it gets to the "VMware Python components" part (which is right near the start), quick rename the entire C:\Program Files\VMware\vCenter Server\python folder.  The install then seems to completely skip over that component as it does not put the C:\Program Files\VMware\vCenter Server\python folder back, so before the install finishes, you need to quick rename C:\Program Files\VMware\vCenter Server\python back again.  This was not a one-off as it took me 6 install attempts to debug what was actually causing this problem.  The sad thing is that if you manually install X:\vCenter-Server\Packages\VMware-python.msi, the files in it are exactly the same as those installed by 6.0U3d so it's not even necessary that this package be reinstalled.  If you run into this problem coming from anything besides 6.0U3d, you may need to manually install X:\vCenter-Server\Packages\VMware-python.msi. I know VMware has effectively given up on the Windows vCenter, but the lack of QA on these releases is astounding.  I mean seriously, when the 3010 error appears, at least give a retry option so you don't have to wait an hour for the entire f'in install to roll back before you can attempt to fix the problem.