Failed consolidation on VCSA "Failed to fetch disk size"

[basicmonkey]

Hello all,

My VCSA has ended up with multiple snapshot orphans following remote backup by Synology Active Backup. There are no snapshots currently listed but still 16x disk children for each of the many VCSA drives. Consolidation doesn't do anything and the UI shows instant completion with no results. Using the 'take snapshot' and then 'delete all' just adds another orphan.

I've tried a backup / re deploy / restore of VCSA and when I do a snapshot and delete, the orphans appear and I can't consolidate again.

The VM log shows this for every consolidation attempt:

2021-03-24T23:22:32.727Z| vmx| W003: Partial data received. fd [209] remaining [8] error [0] [Success]
2021-03-24T23:22:32.727Z| vmx| I005: Unable to receive data from vaai-nasd [Success]
2021-03-24T23:22:32.727Z| vmx| E002: ConsolidateFillSnapDiskTransferArray: Failed to fetch disk size of '/vmfs/volumes/<snip>/vcs7b.basic/vcs7b.basic-000003.vmdk' for consolidate: Unknown error (5)
2021-03-24T23:22:32.727Z| vmx| I005: ConsolidateStart: Failure while populating the snapDataPerCombineOperation array: 'Unknown error' (5)
2021-03-24T23:22:32.727Z| vmx| I005: VigorTransport_ServerSendResponse opID=kmo0vjqp-7151-auto-5io-h5:70001371-c7-f2-daab seq=7572: Completed Snapshot request with messages.
2021-03-24T23:22:32.728Z| vmx| I005: Turning off snapshot info cache.
2021-03-24T23:22:32.728Z| vmx| I005: Turning off snapshot disk cache.
2021-03-24T23:22:32.728Z| vmx| I005: ConsolidateEnd: Snapshot consolidate complete: Unknown error (5).

I've tried consolidation from host via GUI and esxcli. Also while the VCSA is on or off.

VCSA is on local SSD storage, VMFS 6.

I also can't clone or storage vMotion it with an error about the main vmdk file.

This is on vCenter 7.0.2 and ESXI 7.0U1.

Any help would be really appreciated!

Many thanks,

James

[basicmonkey]

Looks like a patch run on the hosts, manually copying to shared NFS storage and re-registering the VM has allowed me to consolidate the VCSA and another poorly VM with lots of orphaned disk deltas. Also updated VAAI Synology plugin too as saw an error there despite not being on NAS.

[chris_arceneaux]

I'm seeing the same issue for multiple VMs, including VCSA, after a recent vSphere upgrade (from 7.0 to update 1). Key thing to note is that I'm using different backup software and I was already using shared storage. This leads me to believe it's a bug in vSphere as the only common thread here is VM snapshots.

vCenter: 7.0.1

ESXi: 7.0 Update 1

As this is a lab, I'm unable to open a support case for this. If someone else sees the same issue and is able, please open a support case. 😀 

[chris_arceneaux]

I'm seeing the same issue for multiple VMs, including VCSA, after a recent vSphere upgrade (from 7.0 to update 1). Key thing to note is that I'm using different backup software and I was already using shared storage. This leads me to believe it's a bug in vSphere as the only common thread here is VM snapshots.

vCenter: 7.0.1

ESXi: 7.0 Update 1

As this is a lab, I'm unable to open a support case for this. If someone else sees the same issue and is able, please open a support case.

[chris_arceneaux]

Quick follow-up:

I'm not seeing any locked files, and my vmware.log shows the same error as the OP.

I powered off the VMs and checked for locked files but came up empty. As my shared storage is NFS, I also looked for "lck" files but there aren't any.

- https://kb.vmware.com/s/article/2107795

- https://kb.vmware.com/s/article/2107795

To temporarily resolve the issue, I restarted management agents on my ESXi servers. Afterwards, consolidation succeeded without issue.

[chris_arceneaux]

Follow-up on this, it's the Synology NFS VAAI vib that's causing the issue. It worked fine with vSphere 7.0 but it apparently introduced this bug manifests in vSphere 7.0 U1. I haven't tested U2 to see if the same behavior still exists.

Removing the vib from all ESXi hosts and then rebooting resolves this issue. Of course, then there's no VAAI support... 😥

[basicmonkey]

Hi Chris. Since updating the Synology VAAI VIB to the latest version, and going to 7.0U2 on the two hosts, I've not seen the issue again.

[chris_arceneaux]

Good to know! Thanks for sharing!

I'm not able to go to U2 yet for other reasons but I'll give this a shot after I upgrade.