Strange error messages in the logs

[profilint]

Hi all,

Please inform if this is not the correct place to ask this type of questionç

Today we have discovered some strange messages in some of the logs, ESXi seems running normally, We do not experience any issues in any of the VMs, But the errors repeat in the logs,

We are running HPE Customized ESXi 7.0.2 Image version 702.0.0.10.7.0 for HPE Servers based on VMware ESXi 7.0.2 release.

Here are the repeating entries in the logs, vmkernel,log and vmkwarnıng,log and syslog

Could someone guide to identify the issue?

Thanks in advance,

Regards,

vmkernel,log:
2022-02-15T11:49:18.286Z cpu26:2097244)ScsiDeviceIO: 4325: Cmd(0x45d8d196b900) 0x28, CmdSN 0x2 from world 2099501 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0
2022-02-15T11:49:18.286Z cpu24:2100111)LVM: 6273: PE grafting failed for device mpx.vmhba33:C0:T0:L0:7, vol 60f1b657-a2dd4ee6-9bfb-b8830351d34e/11237272616595160959: I/O error
2022-02-15T11:49:18.288Z cpu24:2100111)Vol3: 2129: Could not open device 'mpx.vmhba33:C0:T0:L0:7' for volume open: Not found
2022-02-15T11:49:18.288Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 1 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 0 gen 0 :Not found
2022-02-15T11:49:18.288Z cpu24:2100111)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:49:18.288Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 2 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 4 gen 1 :Not found
2022-02-15T11:49:18.659Z cpu26:2097244)NMP: nmp_ThrottleLogForDevice:3796: last error status from device mpx.vmhba33:C0:T0:L0 repeated 20 times
2022-02-15T11:49:18.659Z cpu26:2097244)ScsiDeviceIO: 4325: Cmd(0x45d8d590aa40) 0x28, CmdSN 0x2 from world 2099501 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0
2022-02-15T11:49:18.659Z cpu24:2100111)LVM: 6273: PE grafting failed for device mpx.vmhba33:C0:T0:L0:7, vol 60f1b657-a2dd4ee6-9bfb-b8830351d34e/11237272616595160959: I/O error
2022-02-15T11:49:18.661Z cpu24:2100111)Vol3: 2129: Could not open device 'mpx.vmhba33:C0:T0:L0:7' for volume open: Not found
2022-02-15T11:49:18.661Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 1 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 0 gen 0 :Not found
2022-02-15T11:49:18.661Z cpu24:2100111)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:49:18.661Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 2 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 4 gen 1 :Not found
2022-02-15T11:49:19.025Z cpu26:2097244)ScsiDeviceIO: 4325: Cmd(0x45d8d4a64f40) 0x28, CmdSN 0x2 from world 2099501 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0
2022-02-15T11:49:19.025Z cpu24:2100111)LVM: 6273: PE grafting failed for device mpx.vmhba33:C0:T0:L0:7, vol 60f1b657-a2dd4ee6-9bfb-b8830351d34e/11237272616595160959: I/O error
2022-02-15T11:49:19.027Z cpu24:2100111)Vol3: 2129: Could not open device 'mpx.vmhba33:C0:T0:L0:7' for volume open: Not found
2022-02-15T11:49:19.027Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 1 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 0 gen 0 :Not found
2022-02-15T11:49:19.027Z cpu24:2100111)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:49:19.027Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 2 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 4 gen 1 :Not found

vmkwarning,log:

2022-02-15T11:50:20.992Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:21.353Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:21.714Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:22.075Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:22.436Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:22.797Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:23.158Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:23.519Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:23.880Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:24.244Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:24.605Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:24.966Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0

syslog:
2-15T11:48:22Z root: Call wait loop
2022-02-15T11:48:22Z root: Virtual NIC: in wait loop
2022-02-15T11:48:53Z sfcbd[41121535]: tool_mm_realloc_or_die: memory re-allocation failed(orig=1351200 new=1352000 msg=Cannot allocate memory, aborting
2022-02-15T11:48:53Z root: vusb:
2022-02-15T11:48:53Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:48:53Z root: Call wait loop
2022-02-15T11:48:53Z root: Virtual NIC: in wait loop
2022-02-15T11:48:55Z sfcb-ProviderManager[2100455]: handleSigChld:837734156032 provider terminated, pid=41121535, exit=0 signal=6
2022-02-15T11:49:24Z root: vusb:
2022-02-15T11:49:24Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:49:24Z root: Call wait loop
2022-02-15T11:49:24Z root: Virtual NIC: in wait loop
2022-02-15T11:49:54Z root: vusb:
2022-02-15T11:49:54Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:49:54Z root: Call wait loop
2022-02-15T11:49:55Z root: Virtual NIC: in wait loop
2022-02-15T11:50:00Z crond[2098039]: USER root pid 41122877 cmd /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
2022-02-15T11:50:00Z crond[2098039]: USER root pid 41122878 cmd /bin/crx-cli gc
2022-02-15T11:50:25Z root: vusb:
2022-02-15T11:50:25Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:50:25Z root: Call wait loop
2022-02-15T11:50:25Z root: Virtual NIC: in wait loop
2022-02-15T11:50:56Z root: vusb:
2022-02-15T11:50:56Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:50:56Z root: Call wait loop
2022-02-15T11:50:56Z root: Virtual NIC: in wait loop
2022-02-15T11:51:27Z root: vusb:
2022-02-15T11:51:27Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:51:27Z root: Call wait loop
2022-02-15T11:51:27Z root: Virtual NIC: in wait loop

[DeviVmware]

Is your ESXi running on local datastore ? if yes, do you have local disks connected on path mpx.vmhba33:C0:T0:L0?

[scott28tt]

For next time, "spoilers" are great for dumping logs into a post, to make it easier for scrolling through a thread.

Add a "spoiler" by selecting the triangle icon with exclamation mark on the toolbar.

Here's the difference:

Spoiler

vmkernel,log:
2022-02-15T11:49:18.286Z cpu26:2097244)ScsiDeviceIO: 4325: Cmd(0x45d8d196b900) 0x28, CmdSN 0x2 from world 2099501 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0
2022-02-15T11:49:18.286Z cpu24:2100111)LVM: 6273: PE grafting failed for device mpx.vmhba33:C0:T0:L0:7, vol 60f1b657-a2dd4ee6-9bfb-b8830351d34e/11237272616595160959: I/O error
2022-02-15T11:49:18.288Z cpu24:2100111)Vol3: 2129: Could not open device 'mpx.vmhba33:C0:T0:L0:7' for volume open: Not found
2022-02-15T11:49:18.288Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 1 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 0 gen 0 :Not found
2022-02-15T11:49:18.288Z cpu24:2100111)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:49:18.288Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 2 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 4 gen 1 :Not found
2022-02-15T11:49:18.659Z cpu26:2097244)NMP: nmp_ThrottleLogForDevice:3796: last error status from device mpx.vmhba33:C0:T0:L0 repeated 20 times
2022-02-15T11:49:18.659Z cpu26:2097244)ScsiDeviceIO: 4325: Cmd(0x45d8d590aa40) 0x28, CmdSN 0x2 from world 2099501 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0
2022-02-15T11:49:18.659Z cpu24:2100111)LVM: 6273: PE grafting failed for device mpx.vmhba33:C0:T0:L0:7, vol 60f1b657-a2dd4ee6-9bfb-b8830351d34e/11237272616595160959: I/O error
2022-02-15T11:49:18.661Z cpu24:2100111)Vol3: 2129: Could not open device 'mpx.vmhba33:C0:T0:L0:7' for volume open: Not found
2022-02-15T11:49:18.661Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 1 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 0 gen 0 :Not found
2022-02-15T11:49:18.661Z cpu24:2100111)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:49:18.661Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 2 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 4 gen 1 :Not found
2022-02-15T11:49:19.025Z cpu26:2097244)ScsiDeviceIO: 4325: Cmd(0x45d8d4a64f40) 0x28, CmdSN 0x2 from world 2099501 to dev "mpx.vmhba33:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0
2022-02-15T11:49:19.025Z cpu24:2100111)LVM: 6273: PE grafting failed for device mpx.vmhba33:C0:T0:L0:7, vol 60f1b657-a2dd4ee6-9bfb-b8830351d34e/11237272616595160959: I/O error
2022-02-15T11:49:19.027Z cpu24:2100111)Vol3: 2129: Could not open device 'mpx.vmhba33:C0:T0:L0:7' for volume open: Not found
2022-02-15T11:49:19.027Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 1 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 0 gen 0 :Not found
2022-02-15T11:49:19.027Z cpu24:2100111)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:49:19.027Z cpu24:2100111)Vol3: 4339: Failed to get object 28 type 2 uuid 60f1b658-40b1277a-2259-b8830351d34e FD 4 gen 1 :Not found

vmkwarning,log:

2022-02-15T11:50:20.992Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:21.353Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:21.714Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:22.075Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:22.436Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:22.797Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:23.158Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:23.519Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:23.880Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:24.244Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:24.605Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0
2022-02-15T11:50:24.966Z cpu32:2100357)WARNING: Fil3: 1534: Failed to reserve volume f533 28 1 60f1b658 40b1277a 83b82259 4ed35103 0 0 0 0 0 0 0

syslog:
2-15T11:48:22Z root: Call wait loop
2022-02-15T11:48:22Z root: Virtual NIC: in wait loop
2022-02-15T11:48:53Z sfcbd[41121535]: tool_mm_realloc_or_die: memory re-allocation failed(orig=1351200 new=1352000 msg=Cannot allocate memory, aborting
2022-02-15T11:48:53Z root: vusb:
2022-02-15T11:48:53Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:48:53Z root: Call wait loop
2022-02-15T11:48:53Z root: Virtual NIC: in wait loop
2022-02-15T11:48:55Z sfcb-ProviderManager[2100455]: handleSigChld:837734156032 provider terminated, pid=41121535, exit=0 signal=6
2022-02-15T11:49:24Z root: vusb:
2022-02-15T11:49:24Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:49:24Z root: Call wait loop
2022-02-15T11:49:24Z root: Virtual NIC: in wait loop
2022-02-15T11:49:54Z root: vusb:
2022-02-15T11:49:54Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:49:54Z root: Call wait loop
2022-02-15T11:49:55Z root: Virtual NIC: in wait loop
2022-02-15T11:50:00Z crond[2098039]: USER root pid 41122877 cmd /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
2022-02-15T11:50:00Z crond[2098039]: USER root pid 41122878 cmd /bin/crx-cli gc
2022-02-15T11:50:25Z root: vusb:
2022-02-15T11:50:25Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:50:25Z root: Call wait loop
2022-02-15T11:50:25Z root: Virtual NIC: in wait loop
2022-02-15T11:50:56Z root: vusb:
2022-02-15T11:50:56Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:50:56Z root: Call wait loop
2022-02-15T11:50:56Z root: Virtual NIC: in wait loop
2022-02-15T11:51:27Z root: vusb:
2022-02-15T11:51:27Z root: VIRTUAL NIC:****NO VUSB0 AVAILABLE
2022-02-15T11:51:27Z root: Call wait loop
2022-02-15T11:51:27Z root: Virtual NIC: in wait loop