Read IO performance maybe impacted - after/during Snapshot deletion

[fabian9887]

Hello,

has anyone seen such an event in ESXi?

SESparse: 1012: Read IO performance maybe impacted for disk XXXXXXXX-flat.vmdk: Non-empty delta disk being opened

The VM went offline yesterday and I want to figure out if there's any correlation...?

Seems to occur when removing a NetBackup Snapshot in the vCenter.

Thank you in advance.

Fabian

[AndreasO1]

Hi 

I have the same events on several VM when doing snapshots from synology storage console (application aware)

//Andreas

[Dino_bcs]

Hello.

VMware ESXi, 7.0.3, 19193900 hosts - ethernet [iSER] with Mellanox cards [25Gb]  connected to IBM 7200 FS.

I see the same events: "Read IO performance maybe impacted for disk VM_name-flat.vmdk: Non-empty delta disk being opened"

But we also have problems with host-storage communication[?].

Both support engineers, vmware and storage vendor, told us we should check with the other one. Logs sent, nothing actually helped after their advice to update driver on hosts and some workarounds.

For example: Veeam backup job removing snapshot from random VM and it stuck at 95%, VM become unresponsive, other VMs could be migrated from host, after that "impacted" VM cannot be migrated, host also cannot be put in maintenance/ rebooted etc..

My guess is that vSphere 7U3 still has too many bugs?

[Stefan21]

Hello

VMware ESXi, 7.0.2, 18538813, vCenter 7.02

We are experiencing the same thing as you. Our ESXi then crash at some point and are "not responding".  vMotion and all other vCenter functions can no longer be run on the ESXi, connection to vCenter disconnected.

VMware Support is at a loss, spinning in circles. Storage manufacturer can prove that the storage has no bottlenecks or errors.

09.03.2022 12:18:07 Cannot synchronize host vh19.domain.local. An error occurred while communicating with the remote host.
09.03.2022 12:18:07 Alarm 'Host memory usage' on vh19.domain.local changed from Green to Gray
09.03.2022 12:18:07 Alarm 'Host CPU usage' on vh19.domain.local changed from Green to Gray
09.03.2022 12:18:07 Cannot synchronize host vh19.domain.local. An error occurred while communicating with the remote host.
09.03.2022 12:18:07 Alarm 'Host connection and power state' on vh19.domain.local changed from Green to Red
09.03.2022 12:18:07 Host vh19.domain.local in [Datacenter name] is not responding
09.03.2022 12:15:28 root User root@127.0.0.1 logged out (login time: Wednesday, 09 March, 2022 11:15:28 AM, number of API invocations: 7, user agent: pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64))
09.03.2022 12:15:28 root User root@127.0.0.1 logged in as pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64)
09.03.2022 12:14:59 root User root@127.0.0.1 logged out (login time: Wednesday, 09 March, 2022 11:14:59 AM, number of API invocations: 7, user agent: pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64))
09.03.2022 12:14:59 root User root@127.0.0.1 logged in as pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64)
09.03.2022 12:14:44 root User root@127.0.0.1 logged out (login time: Wednesday, 09 March, 2022 11:14:44 AM, number of API invocations: 3, user agent: pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64))
09.03.2022 12:14:44 root User root@127.0.0.1 logged in as pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64)
09.03.2022 12:12:53 root User root@127.0.0.1 logged out (login time: Wednesday, 09 March, 2022 11:12:53 AM, number of API invocations: 7, user agent: pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64))
09.03.2022 12:12:53 root User root@127.0.0.1 logged in as pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64)
09.03.2022 12:12:20 root User root@127.0.0.1 logged out (login time: Wednesday, 09 March, 2022 11:12:20 AM, number of API invocations: 7, user agent: pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64))
09.03.2022 12:12:20 root User root@127.0.0.1 logged in as pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64)
09.03.2022 12:11:21 root User root@127.0.0.1 logged out (login time: Wednesday, 09 March, 2022 11:11:21 AM, number of API invocations: 7, user agent: pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64))
09.03.2022 12:11:21 root User root@127.0.0.1 logged in as pyvmomi Python/3.8.3 (VMkernel; 7.0.2; x86_64)
09.03.2022 12:07:33 HostSystem SSH session was opened for 'root@192.118.101.91'.
09.03.2022 10:50:58 HostSystem com.vmware.vim.eam Agent status changed from yellow to green
09.03.2022 10:50:58 HostSystem com.vmware.vim.eam Agent status changed from green to yellow
09.03.2022 09:37:32 System Hardware Sensor Status: Processor Green, Memory Green, Fan Green, Voltage Green, Temperature Green, Power Green, System Board Green, Battery Green, Storage Green, Other Green
09.03.2022 03:36:35 System Hardware Sensor Status: Processor Green, Memory Green, Fan Green, Voltage Green, Temperature Green, Power Green, System Board Green, Battery Green, Storage Green, Other Green
08.03.2022 23:24:35 HostSystem Read IO performance maybe impacted for disk xxxx_1-flat.vmdk: Non-empty delta disk being opened
08.03.2022 23:24:28 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 23:01:00 HostSystem Read IO performance maybe impacted for disk xxxx_1-flat.vmdk: Non-empty delta disk being opened
08.03.2022 23:00:56 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:50:41 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:47:24 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:47:18 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:45:59 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:44:25 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:44:19 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:43:17 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:35:00 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened
08.03.2022 22:34:58 HostSystem Read IO performance maybe impacted for disk xxxx-flat.vmdk: Non-empty delta disk being opened

[RegionalEye]

Have you had any other issues with Synology?  I'm running a uc3200 and having these errors, and random spikes in latency that bring us to a halt in some cases, prompting VMware to warn that connectivity to the storage has been lost.  Are you using v7?

[RegionalEye]

Have you found any solution or explanation on this?

I experienced this as well with one of my hosts, "ESXi then crash at some point and are "not responding""

[Stefan21]

Unfortunately not, the VMware support case is still unchanged.

[RegionalEye]

I've upgraded to the latest build 19193900 today after this happened to me again overnight.  Will let you know how stability goes.

[tonyanshe]

@RegionalEye just reaching out to query if your issue was resolved on build 19193900?

We have been experiencing similar issues as the users in this thread with our ESXi hosts crashing, sometimes just a few VMs other times all VMS on the hosts. A hard reset was the only way to recover the host.

This looks to correlate to when we upgraded to ESXi 7.0 U2 build - 18538813 so last week I upgraded to ESXi 7.0 U3 build - 19482537.

I since came across this thread so I am hoping your hosts has stabilized since running ESXi 7.0 U3 build - 19193900

@Dino_bcs @Stefan21 @AndreasO1 did you come across any solution - we also have had multiple VMware, storage and hardware vendors tickets opened, to no avail. 

[Dino_bcs]

Hello,

well VMware support advise us to upgrade to 7.0.3d (Esxi hosts), we did and problem is resolved.

@tonyanshe  You can try with only one host but it looks stable, running well, hope it will solve yr issues..

✌️