This event indicates that the ESX host's connectivity to the volume (for which this event was generated) degraded due to the inability of the host to renew its heartbeat for period of approximately 16 seconds (the VMFS lock breaking lease timeout). After the periodic heartbeat renewal fails, VMFS declares that the heartbeat to the volume has timed out and suspends all I/O activity on the device until connectivity is restored or the device is declared inoperable.
There are two components to this:
A host indicates its liveness by periodically (every 3 seconds) performing I/O to its heartbeat on a given volume. Therefore, if no activity is seen on the host's heartbeat slot for a period of time, then we can conclude that the host has lost connectivity to the volume. This wait time is a little over 5 heartbeat intervals or 16 seconds to be precise. Example
If an ESX host has mounted a volume san-lun-100 from device naa.60060160b4111600826120bae2e3dd11:1 and loses connectivity (due to a cable pull, disk array failure, and so on) to the device for a period exceeding 16 seconds, the following error message appears:
Lost access to volume 496befed-1c79c817-6beb-001ec9b60619 (san-lun-100) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.
Impact
All I/O, metadata operations to the specific volume from COS, user interface (vSphere Client), or virtual machines are internally queued and retried for some duration of time. If the volume or storage device connectivity is not restored within that duration of time, such I/O operations fail. This might have an impact on already running virtual machines as well as any new power on operations by virtual machines.
Solution
To resolve this issue:
- Connect to the vCenter Server using vSphere Client.
- Select the Storage View tab to map the HBA (Host Bus Adapter) associated to the affected VMFS volume.
- Follow the steps provided in Troubleshooting fibre channel storage connectivity (1003680) to identify and resolve the path inconsistencies to the LUN.
- If connections are restored, VMFS automatically recovers the heartbeat on the volume and continues the operation.
To resolve this issue using the service console:
- Connect to the ESX host’s service console.
- Run the following commands:
- Query VMFS datastore properties.
Example:
# vmkfstools –P san-lun-100
File system label (if any): san-lun-100
Mode: public
Capacity 80262201344 (76544 file blocks * 1048576), 36768317440 (35065 blocks) avail
UUID: 49767b15-1f252bd1-1e57-00215aaf0626
Partitions spanned (on "lvm"): naa.60060160b4111600826120bae2e3dd11:1
- Use esxcfg-mpath along with the naa ID of the LUN (Logical Unit Number) output from the above command to identify the state of all the paths to affected LUN.
Example:
# esxcfg-mpath -b -d naa.60060160b4111600826120bae2e3dd11
naa.60060160b4111600826120bae2e3dd11 : DGC Fibre Channel Disk (naa.60060160b4111600826120bae2e3dd11) vmhba0:C0:T0:L0 LUN:0 state:active fc Adapter:
WWNN: 20:00:00:00:c9:7d:6c:e0 WWPN: 10:00:00:00:c9:7d:6c:e0 Target: WWNN: 50:06:01:60:b0:22:1f:dd WWPN: 50:06:01:60:30:22:1f:dd vmhba0:C0:T1:L0 LUN:0 state:standby fc Adapter:
WWNN: 20:00:00:00:c9:7d:6c:e0 WWPN: 10:00:00:00:c9:7d:6c:e0 Target: WWNN: 50:06:01:60:b0:22:1f:dd WWPN: 50:06:01:68:30:22:1f:dd
- Follow the steps provided in Troubleshooting fibre channel storage connectivity (1003680) to identify and resolve the path inconsistencies to the LUN.
- If connections are restored, VMFS automatically recovers the heartbeat on the volume and continues the operation.
Note: For additional information, see Troubleshooting LUN connectivity issues on ESXi/ESX hosts (1003
Please mark my Answer correct and like if it helped. Thanks
