ESXi 7.0u3 missing system sensors after reboot

[vmdetex]

We have a Lenovo SR630 running the Lenovo customized version of ESXi 7.0u3.  The system sensors were displaying properly until a recent reboot where it now only shows "No sensor data available". No changes were made to the server prior to the reboot.  I have make sure CIM service is running.  What would cause them to disappear after a reboot?

[vmdetex]

Here's the ipmi related entries in vmkernel.log

2022-04-13T17:22:57.640Z cpu3:2097585)Activating Jumpstart plugin ipmi.
2022-04-13T17:22:57.698Z cpu31:2097788)Loading module ipmi ...
2022-04-13T17:22:57.700Z cpu31:2097788)Elf: 2119: module ipmi has license VMware
2022-04-13T17:22:57.739Z cpu31:2097788)ipmi: ACPI IPMI Entry: Address: 0xcc0, System Interface: 0, Map Type: 0
2022-04-13T17:22:57.739Z cpu31:2097788)ipmi: SMBIOS IPMI Entry: Address: 0xcc0, System Interface: 0, Alignment: 1, Map Type: 0
2022-04-13T17:22:57.739Z cpu31:2097788)ipmi: KCS Port Map: Command Port: 0xcc1 Data Port: 0xcc0
2022-04-13T17:22:57.743Z cpu31:2097788)Mod: 4821: Initialization of ipmi succeeded with module ID 58.
2022-04-13T17:22:57.743Z cpu31:2097788)ipmi loaded successfully.
2022-04-13T17:22:57.751Z cpu4:2097585)Jumpstart plugin ipmi activated.
2022-04-13T17:23:11.968Z cpu0:2099990)SchedVsi: 2083: Group: host/vim/vmvisor/plugins/vmw_ipmi(18763): max=100 min=100 minLimit=unlimited shares=1000, units: mb
2022-04-18T18:10:37.927Z cpu27:2176085)SchedVsi: 1017: Group vmw_ipmi could not be created: Already exists
2022-04-18T18:50:45.269Z cpu0:2177190)SchedVsi: 2083: Group: host/vim/vmvisor/plugins/vmw_ipmi(369801): max=100 min=100 minLimit=unlimited shares=1000, units: mb
TSC: 378856 cpu0:1)BootConfig: 711: ipmiEnabled = TRUE (1)
2022-04-18T23:17:45.618Z cpu27:2097564)Activating Jumpstart plugin ipmi.
2022-04-18T23:17:45.676Z cpu5:2097791)Loading module ipmi ...
2022-04-18T23:17:45.676Z cpu5:2097791)Elf: 2119: module ipmi has license VMware
2022-04-18T23:17:45.715Z cpu5:2097791)ipmi: ACPI IPMI Entry: Address: 0xcc0, System Interface: 0, Map Type: 0
2022-04-18T23:17:45.715Z cpu5:2097791)ipmi: SMBIOS IPMI Entry: Address: 0xcc0, System Interface: 0, Alignment: 1, Map Type: 0
2022-04-18T23:17:45.715Z cpu5:2097791)ipmi: KCS Port Map: Command Port: 0xcc1 Data Port: 0xcc0
2022-04-18T23:17:48.127Z cpu4:2097791)WARNING: ipmi: WriteStartPhase:227: ipmi: Timed out waiting for IBF to clear in write start phase.. Error: Timeout
2022-04-18T23:17:48.127Z cpu4:2097791)WARNING: ipmi: SysIntKcs_ProcessRequest:578: ipmi: Failure in KCS write start phase. Error: Timeout
2022-04-18T23:17:48.127Z cpu4:2097791)WARNING: ipmi: GetDeviceId:779: ipmi: Failed to process Get Device ID request. Error: Timeout
2022-04-18T23:17:50.506Z cpu4:2097791)WARNING: ipmi: WriteStartPhase:217: ipmi: Timed out waiting for IBF to clear in write start phase.. Error: Timeout
2022-04-18T23:17:50.506Z cpu4:2097791)WARNING: ipmi: SysIntKcs_ProcessRequest:578: ipmi: Failure in KCS write start phase. Error: Timeout
2022-04-18T23:17:50.506Z cpu4:2097791)WARNING: ipmi: GetDeviceId:779: ipmi: Failed to process Get Device ID request. Error: Timeout
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: WriteStartPhase:217: ipmi: Timed out waiting for IBF to clear in write start phase.. Error: Timeout
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: SysIntKcs_ProcessRequest:578: ipmi: Failure in KCS write start phase. Error: Timeout
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: GetDeviceId:779: ipmi: Failed to process Get Device ID request. Error: Timeout
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: IpmiDriver_Init:232: ipmi: IPMI Device failed to respond to the GET DEVICE ID request. Error: Timeout
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: CreateIpmiDrivers:1246: ipmi: Failed to initialize IPMI driver. Error: Timeout
2022-04-18T23:17:52.929Z cpu4:2097791)ipmi: No valid IPMI devices were discovered based upon PCI, ACPI or SMBIOS entries, attempting to discover IPMI devices at default locations
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: KcsRegPort_Init:95: ipmi: Error registering IO resource for KCS data port address at 0xca2.. Error: Bad address range
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: IpmiSysIntKcs_Init:769: ipmi: Failure to inialize KCS registers. Error: Bad address range
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: IpmiDriver_Init:219: ipmi: Failed to initialize IPMI system interface. Error: Bad address range
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: CreateIpmiDrivers:1246: ipmi: Failed to initialize IPMI driver. Error: Bad address range
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: SanityCheckRegs:111: ipmi: Reading the BT Control Register produced an invalid value: 0xFF
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: IpmiSysIntBt_Init:107: ipmi: Failed to initialize BT registers. Error: Failure
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: IpmiDriver_Init:219: ipmi: Failed to initialize IPMI system interface. Error: Failure
2022-04-18T23:17:52.929Z cpu4:2097791)WARNING: ipmi: CreateIpmiDrivers:1246: ipmi: Failed to initialize IPMI driver. Error: Failure
2022-04-18T23:17:52.929Z cpu4:2097791)ipmi: Failed to create any IPMI drivers
2022-04-18T23:17:52.929Z cpu4:2097791)ipmi failed to load.
2022-04-18T23:17:52.930Z cpu4:2097791)WARNING: Elf: 3139: Kernel based module load of ipmi failed: Failure <Mod_LoadDone failed>
2022-04-18T23:17:52.938Z cpu16:2097564)Jumpstart plugin ipmi activation failed: ipmi->start() failed: exited with code 1
2022-04-18T23:17:59.062Z cpu0:2099993)SchedVsi: 2083: Group: host/vim/vmvisor/plugins/vmw_ipmi(18826): max=100 min=100 minLimit=unlimited shares=1000, units: mb
2022-04-18T23:31:41.372Z cpu29:2101501)SchedVsi: 1017: Group vmw_ipmi could not be created: Already exists
TSC: 372920 cpu0:1)BootConfig: 711: ipmiEnabled = TRUE (1)

[bazanovv]

For me with same ESXi 7.0u3 (Lenovo custom image) and Lenovo SR630 / SR650 servers sensors begin to work after executing command on ESXi to enable CIM service properly

esxcli system wbem set -e true

and rebooting ESXi host.

P.S. GUI in vCenter sometimes shows wrong / old sensors data. You have to refresh it with the ctrl-F5 key to get current state.

[bazanovv]

Another thing to look at is in the what's new in 7.0 update 3d release:

• PR 2878701: In the VMware Host Client, you see an error that no sensor data is available

  In the VMware Host Client, you see the error No sensor data available due to an issue with the DateTime formatting. In the backtrace, you see logs such as:
  hostd[1051205] [Originator@6876 sub=Cimsvc] Refresh hardware status failed N7Vmacore23DateTimeFormatExceptionE(Error formatting DateTime).

  This issue is resolved in this release.

[vmdetex · Accepted solution]

The issue has been resolved. The option "IPMI over KCS Access" in the XClarity BMC Configuration Security settings had to be disabled, then re-enabled. After doing this all of the sensors returned.

[PePierias]

We have the same issue on some Thinksystems 650 (latest firmware). Should "IPMI over KCS Access" be disabled/re-enabled while vmware is running, or when the system is shutdown?