eode Tracker

Re: Horizon Client and Wandering Wifi LLC

eode — Tue, 31 Oct 2023 22:46:10 GMT

Hi! I noticed the very same while upgrading my Horizon Client (on macOS) to version "2309" (8.11.0-22660933) today. Based on my "preliminary research" it seems to be an AirWatch-related telemetry-services, now shipping with the Horizon Client

I also noticed some new background services in macOS (which initially got me curious). After some research, I successfully uninstalled the (new) telemetry services, and they are not listed under System Settings(.app) > General > Login Items > Allow in the Background anymore.

com.vmware.deem?

Guestimate: Digital Employee Experience Management

Based on https://www.vmware.com/nordics/products/workspace-one/digital-employee-experience-management.html

com.vmware.vmwetlm?

Guestimate: VMware Experience Management Service

Based on https://docs.vmware.com/en/VMware-Workspace-ONE-Intelligence/services/intelligence-documentation/GUID-IntelExperienceManagement.html

Please note that I have not investigated any side affects (yet). I wrote a quick blog post/summary on findings (and how to uninstall) here: https://espenodegaard.com/blog/2023-10-31-horizon-client-2309-wandering-wifi-lcc

Re: An error occurred while saving the snapshot

eode — Tue, 01 Jun 2021 06:52:19 GMT

If you see my first reply, I'm getting error with disk lock, and from the process DISKLIB-CBT. Workaround to clear lock for me; migrate the VM to another host. I have not had any issues with locking since. May be issues with CBT/CTK, which also may be disabled on VM level (if only occuring on some VMs, etc.). May be other underlying issues with hosts, etc.

Previous post
https://communities.vmware.com/t5/VMware-vSphere-Discussions/An-error-occurred-while-saving-the-snapshot/m-p/2847377/highlight/true#M39149

The error I'm referring to

2021-05-16T15:38:49.953Z| vmx| | W003: DISKLIB-CBT : ChangeTrackerESX_GetMirrorCopyProgress: Failed to copy mirror: Lost previously held disk lock

Since this process fail, the following snapshot request process (SnapshotPrepareTakeDoneCB) will also fail, hence the error seen in vSphere Client (my guess)

2021-05-16T15:38:49.976Z| vmx| | I005: SNAPSHOT: SnapshotPrepareTakeDoneCB: Failed to prepare block track. 2021-05-16T15:38:49.976Z| vmx| | I005: SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (Could not get mirror copy status).

I wrote a short guest post, which includes the full vmware.log for a VM, while having snapshot issues, and trying to trigger a manual snapshot (regular snapshot, via vSphere Client), which includes some more details. The relevant part is of course the warning from DISKLIB-CBT, regarding the lock.

https://vninja.net/2021/05/18/error-occurred-while-saving-snapshot-msg.changetracker.mirrorcopystatus

I would start with verifying if it's the same error (see the same error in the vmware.log), then go from there. If Production w/SnS, just open a SR with VMware, OFC.

Re: An error occurred while saving the snapshot

eode — Mon, 17 May 2021 06:49:36 GMT

Interesting, I'll have to wait to see if the same pattern happens here. Backup went fine yesterday, and this night (as usual). Only occured on one (1) VM for me, so far.

- I did also notice a DRS vMotion (auto) occured on the VM earlier the same day (some hours before I got the snapshot/locking issues).

Q: Maybe you could check, if same pattern occured in your environment; e.g. vMotions/changes on the VM before snapshot issues occured? If so, lowering the DRS threshold could also help "slow down" the occurence (not a permanent solution, but the impact/frequency may be reduced).

I'm also running ESXi 7.0.2 build-17867351, in a 4-node vSAN cluster.

Q: Are you also running vSAN, or is this regular VMFS/NFS datastore?

Re: An error occurred while saving the snapshot

eode — Sun, 16 May 2021 15:58:00 GMT

Hi!

Got the same error today, basically triggered from Veeam, but related to taking snapshot of my VM, before backup.

Finding
I did a quick troubleshoot in the vmware.log of the affected VM, and found a locking issue, like this:
"2021-05-16T15:38:49.953Z| vmx| | W003: DISKLIB-CBT : ChangeTrackerESX_GetMirrorCopyProgress: Failed to copy mirror: Lost previously held disk lock".

Resolution/workaround
Since the locking is held locally by the host, I just did a vMotion of the VM (to another host, in my cluster), to re-issue the lock by another host. Re-trying snapshot, it now completed successfully (and backup is now working again).

Re: View saved vSAN Observer Logs

eode — Fri, 04 Jan 2019 13:21:00 GMT

Yup, try opening the vSAN Observer bundle from a computer with internet access. The graphs.html and stats.html have some references to scripts e.g. to https://code.jquery.com (you'll find this running e.g. Chrome in "Developer Mode", and look for what's "stuck on loading", e.g. "connecting to netdns.bootstrapcdn.com"). If you have internet access, maybe some of the are blocked? (you'll find out when running in "devmode"). If I remember correctly, boostrap.min.js also have some additional references (you'll see this when accessing the "Disks Deep-Dive view").

If you don't have internet access, you may also download the scripts, and do some quick & dirty "hacks" on the vSAN Observer pages (e.g. directly in the graphs.html and stats.html), putting the necessery files somewhere available - them the pages will load just fine. Both works when files available (e.g. vSAN Observer directly from vCSA and/or exported bundle). Would be nice if files where included directly in vSAN Observer-bundle, but hey; guessing this just is a tool for VMware / technical geeks now days :smileycool:.

Good luck!

Best regards,

Espen Ødegaard

Re: vSAN PSOD - emulex NIC ?

eode — Fri, 09 Mar 2018 11:14:35 GMT

Hi.

Not the exact same error (2682 vs 2688), but yeah, it's the elxnet_getRxPfragInfo, for sure.

Just to "verify", I'm guessing you're running the same (updated) firmware as well (latest on VCG)? You could check with this command (look for "Firmware Version")

esxcli network nic get -n vmnic5 | grep Version

Note: Where "vmnic5" is your vmnic-number. Also; case sensitive on "Version" (or you could just use grep -i version)

Also, I notice different ESXi-build. As mentioned I've not seen this on 6.0 U3 with Patch 6, that's ESXi build 6921384 - after applying the firmware & driver update. I did however see the issue before upgrading fimware & driver (on the very same ESXi build), and also on ESXi build 5572656 (before upgrading firmware & driver).

Curious; does this happen "out of the blue", or while triggering something (in our case, we did see network latency, but PSOD first occured while putting the host in maintenanceMode).

Re: vSAN PSOD - emulex NIC ?

eode — Mon, 26 Feb 2018 15:04:01 GMT

Hi.

We experienced the same PSOD (elxnet_getRxPfragInfo - 2668) on some ProLiant DL380 Gen9 (the 556FLR-SFP+ Adapter being the similar factor here), running on the 11.1.145.0 / 11.1.183.62 combination (it's on VMware VCG, but not the latest version). Mainly, we started our "investigation" based on a latency issue, but the PSOD occured at a later stage, when we put hosts in maintenance mode. As the vmkernel/logs was not reporting any specific errors, we decided to upgrade the driver/firmware combination, pending verification from both VMware GSS & HPE. We have not seen the issue since upgrading the driver & firmware (this was in late December, 2017).

Both VMware & HPE finished their analysis, and returned with the very same action plan which we had already performed (upgraded driver & firmware). HPE also recommended a specific ESXi-build, but we were already on a newer version.

In short:

ESXi-build: 6921384 (aka 6.0 U3 + Patch 6)
Combination when issue was active: 11.1.145.0 / 11.1.183.62
Combination when issue was resolved: 11.2.1149.0 / 11.2.1226.20

So, quick questions:

Which version of ESXi are you running, both before & after upgrade?
Which version of driver (elxnet-module), both before & after upgrade?
Which version of firmware (before & after upgrade)?

Just for reference, the SVID:SDID on the 556FLR-SFP+-device is 103c:220a.

Regards,

Espen Ødegaard

Re: High "Frontend Read Latency" on DiskGroup

eode — Mon, 04 Dec 2017 14:40:24 GMT

Hi.

Was waiting for my SR to be closed before replying here; this is a bug in reporting. Same "latency" was seen in both Web Client (under %host% > Monitor > Performance > Virtual SAN - Disk Group > "Frontend(Guest) Latency", and in VSAN Observer (on LSOM/lsomhost, that's VSAN Disks (deep-dive) > %host% > Host disk-layer aggregate stats > Full graphs > Look in the first graph - latency). Value was reporting around 200.00 - 500.00, when it should have been 0.2 - 0.5.

Good news is that it's resolved in the latest 6.0 U3-release (build 6921384).

From the release notes / VMware Knowledge Base

"The Host disk-layer aggregate stats full graphs view under the vSAN Disk (deep-dive) tab might report wrong latency values, because the latency values are generated in nanoseconds, but wrongly interpreted as milliseconds."

Nanoseconds.. well.. based on the numbers (500 to 0.5) it's microseconds. Anyways, it's now fixed! :-).

Re: How to migrate thick VMs from traditional storage to thin in VSAN?

eode — Fri, 16 Jun 2017 08:58:29 GMT

Question

On what findings are you basing your conclusion that it's not thin? What do you see? What are the numbers?

Short answer

In short; if OSR = 0 in the storage policy, then OSR should be 0. No need to "select" thin or thick - this is all in policy. You could also verify this in RVC (se previous comments).

Other possible "solution" - disk reclaim

Regarding disk reclaim (vSphere "used space" dosn't match OS used space)

If you're looking at the "GB used storage space" (on the disk, in Web Client), the "used storage space" should usually equal the OS space * policy (e.g. FTT=1, then it's OS disk used space * 2). If calculations does not match (OS used vs "GB used storage space"), you could reclaim diskspace, e.g. using "zeropunch" on the disk (using SDelete from Sysinternals in Windows (-z) or dd in Linux). This needs to be done before migrating the data (or you need to migrate back and forth, again). Usually with datamover, null blocks are not "reclaimed".

Note: This will fill up the virtual disk (then "zap the zero's, sort of speaking"), so make sure you have enough underlying space for this operation (if thin disks).

After the disk is "filled with zero-blocks" (Or "zapped!" - in SDelete), you should be able to shrink the VMDK (removes the zero-blocks). Sadly, this isn't supported on vSAN (yet), so you have to Storage vMotion the VMDK. If a new blocksize on the target, the "zero-blocks" will *not* be copied, so in short, only the data will be moved.

If you're not able to use Storage vMotion, you could also move the VMDK to a VMFS DS, and run vmkfstools -K /path/to/disk/vm.vmdk, which will "punch out the zero's" (this operation will requires you to unlock the VMDK, which usually means shutting down the VM - or detach the disk).

After migration (or --punchzero), if you recheck you're "GB used storage space", it should now equal the OS space * policy.

Also see KB2004155 from VMware:

Storage vMotion to thin disk does not reclaim null blocks

https://kb.vmware.com/kb/2004155

Best regards,

Espen Ødegaard

Re: Intel DC P3600 NVMe HCL Issue

eode — Fri, 23 Sep 2016 08:46:38 GMT

As far as I understand, this is a bug in the health check-plugin.

The VID, DID, SVID and SSID is now all matching the updated online HCL (from my P3700-post), but the health check still throws a warning on the P3700 AIC. Got confirmed from VMware (SR) that this would be fixed in future releases of the health check-plugin (still wating...).

FYI: Using the 1.2.0.27-4vmw/8DV10171 combination here.

Re: Intel DC P3700 Firmware

eode — Wed, 15 Jun 2016 11:59:52 GMT

Quick update, regarding VSAN HCL DB:

Got confirmed in our SR that the "Warning on the P3700 HHHL AIC" was due to a VSAN Health-plugin issue, and could be ignored (will be fixed in the following health releases).

Regarding firmware-version:

Was also told to downgrade the stock firmware (8DV10171) to 8DV10131. Hopefully the VSAN HCL DB will be updated shortly (based on Intel's response from yesterday, regarding the *171-firmware, it should be verified/added by VMware next week). Wondering about VMware's updated recommendation on driver (with the new firmware).

Re: Intel DC P3700 Firmware

eode — Fri, 27 May 2016 04:03:49 GMT

Yes, that's my thoughts as well. I've commented this in my SR w/VMware. If lucky, the VSAN HCL "DB" (JSON-file) will be corrected (unless I'm misunderstanding the logic of the HCL-check).

Re: Intel DC P3700 Firmware

eode — Thu, 26 May 2016 12:47:50 GMT

May I ask - which model/type of the Intel DC P3700 are you guys running? While searching through the VSAN HCL DB - the "cool way" (JSON-file directly - http://partnerweb.vmware.com/service/vsan/all.json‌‌) - I've found that the SSID of the SSDPE2MD800G4 (800GB, 2,5-inch) is listed as SSID 3703, and the SSDPEDMD800G4 (800GB, HHHL AIC) also has SSID 3703. Our DC P3700, 800GB, HHHL AIC has SSID of 3702, not 3703, which probably is the reason why the Health Check gives us a "Warning" (does not match any SSID. Different driver or firmware will in that case give the same result, as it still doesn't match any SSID).

Regarding identical SSID in the HCL

A quick search in the JSON-file, and you'll find the following relevant IDs (output from today - this may change):

"id": 39653,
"model": "Intel SSD DC P3700 Series SSDPE2MD800G4 (800 GB, 2.5-inch)",
"vid": "8086",
"did": "0953",
"svid": "8086",
"ssid": "3703",

"id": 39659,
"model": "Intel SSD DC P3700 Series SSDPEDMD800G4 (800 GB, HHHL AIC)",
"vid": "8086",
"did": "0953",
"svid": "8086",
"ssid": "3703",

Checking Vendor ID (VID), Device ID (DID), Sub-Vendor ID (SVID) and Sub-Device ID (SSID)

vmkchdev -l |grep vmhba4

0000:84:00.0 8086:0953 8086:3702 vmkernel vmhba4

Regarding driver & firmware-versions

Our NVMe-device is also shipped with FW 8DV10171 (verified with the Intel DCT).

Based on our VID, DID, SVID and SSID for the device, the following HCLs is available:

From the "General HCL"-list (cat=io): FW 8DV10171 & nvme version 1.2.0.27-4vmw (VMware Async)

https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=38463

From the "VSAN HCL" (cat=ssd): FW 8DV10131 & nvme 1.0e.1.1-1OEM.550.0.0.1391871 (which actually is "intel-nvme", as this is only available as Partner Async-driver, as far as I know).

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=ssd&productid=39653

Still waiting for a response on our SR - just wanted to let you know our findings (in case it helps).

Best regards,

Espen Ødegaard

Re: vmware-dataservice-sca status changed from yellow to green

eode — Tue, 01 Mar 2016 15:16:21 GMT

Adding the "vCenter Server managed address" in the "Runtime settings" seems to have solved the issue for me as well.

That's Web Client > vCenter > Manage > Settings > General > Edit > Update "Runtime settings". Reboot VC-server.

PS: Still running the latest VCSA (U1b / 3339084).

Br.

Espen

Re: vmware-dataservice-sca status changed from yellow to green

eode — Fri, 15 Jan 2016 10:43:35 GMT

I know. Bummer. Actually seemed fine after the U1b-upgrade (I upgraded about 5 days ago), until last night, then the alarms hit me (lots of email notifications). Seems like my little "dummy_path"-script doesn't apply as a workaround this time (even though the errors disappears from the sca.log). Removed the SendEmail-action for now. Maybe I'll dig into the VCSA logs later.

Re: vmware-dataservice-sca status changed from yellow to green

eode — Fri, 08 Jan 2016 14:16:15 GMT

If running VC in Windows, you could check the sca.log from:

%allusersprofile%\vmware\vcenterserver\logs\sca\sca.log, which is usually C:\programdata\vmware\vcenterserver\logs\sca\sca.log

Like others, if it's regarding the "dummy_path"-script, this (hopefully) is a known issue by now. As commented above. I've created a "dummy_path"-script (which will only exit, with the status 0 see my comment above). I've not tested this in VC in Windows, but guessing it's the same issue (script not found = script not found, right?).

Before, I was getting alarms from "Data Service" every 1-2 minutes. If really quick, I saw something like "The Data Service container process is running low on heap memory (> 90% utilized.)". I've also traced the sca-process (java), just to monitor the memory usage.

The alarms haven't triggered since I created the "dummy_path"-script.

Re: Data Service Container Process Heap Memory Alarms

eode — Thu, 31 Dec 2015 02:35:53 GMT

FYI: I've also posted a reply here:

Re: vmware-dataservice-sca status changed from yellow to green

Gave this a look tonight. Seems like the vmware-dataservice-sca/vmware-sca is looking for a script called "dummy_path" (I know).

Found this little gem in the sca.log:

grep -Hin "dummy" /var/log/vmware/sca/sca.log|head -1

/var/log/vmware/sca/sca.log:8:2015-12-30T23:17:36.083Z [pool-5-thread-14 ERROR com.vmware.sca.servicecontrol.handlers.ScriptControlHandler] Failed to locate script; name: dummy_path, dirs: [/usr/lib/vmware-sca/scripts, /etc/vmware-sca/scripts], [java.lang.Exception: Script doesn't exist; path: /etc/vmware-sca/scripts/dummy_path]

So created a little "dummy_path"-script, which will only exit when run.

cat /etc/vmware-sca/scripts/dummy_path

#!/bin/sh

#I'm dumb, and will now exit. With a zero. Because. Awesome.

exit 0

I didn't have the time to check which process which was trying to run the script, so I gave everyone rights (I know. Not cool.).

chmod 777 /etc/vmware-sca/scripts/dummy_path

Profit (I've not seen the issue since, nor any complaints about the dummy_path-script). Oh, and I'm running the VCSA 6.0.0.10000 Build Number 3018521.

Guessing this isn't a "supported workaround", or anything like that (and may cause other problems). Just playing around with my home lab.

Regards,

Espen Ødegaard

Re: vmware-dataservice-sca status changed from yellow to green

eode — Thu, 31 Dec 2015 02:31:59 GMT

Gave this a look tonight. Seems like the vmware-dataservice-sca/vmware-sca is looking for a script called "dummy_path" (I know).

Found this little gem in the sca.log:

grep -Hin "dummy" /var/log/vmware/sca/sca.log|head -1

So created a little "dummy_path"-script, which will only exit when run.

cat /etc/vmware-sca/scripts/dummy_path

#!/bin/sh

#I'm dumb, and will now exit. With a zero. Because. Awesome.

exit 0

I didn't have the time to check which process which was trying to run the script, so I gave everyone rights (I know. Not cool).

chmod 777 /etc/vmware-sca/scripts/dummy_path

Profit (I've not seen the issue since, nor any complaints about the dummy_path-script). Oh, and I'm running the VCSA 6.0.0.10000 Build Number 3018521.

Guessing this isn't a "supported workaround", or anything like that (and may cause other problems). Just playing around with my home lab.

Regards,

Espen Ødegaard