ESXi-7.0U2a VM hangs when creating snapshot with Veeam

[BohdanKotelyak]

Hi All,

I've a problem. Just installed new server with ESXi-7.0U2a (latest build).

I moved 4 VM's from old server to this new one.

Than i configured Veeam to make backups.

Every time when Veeam starting create backup, the VM(every time it's different VM) the VM become unavailable.

It task status window in Vmware web client i see "Creating VM Snapshot 0%" and it hangs.

I'm not able to access VM at that time, not able no restart it, not able even to reset. The only way is to reset the host.

Creating VM Snapshot 0% hanging even when i stop Veeam job for backup, and even when i restart the server where veeam is installed.

I have the same problem on two of my clients .

I'm using the last version of Esxi and Veeam.

P.S.

This Veeam is making backups from another host with Vmware 6.7 without any problems.

Does anyone have any ideas?

[scott28tt]

1. What happens if you take a snapshot of the same VM manually?

2. Have you reported this on the Veeam forum?

[BohdanKotelyak]

Thank you for the reply.

Unfortunattely i cannot try making snapshots manually after i know it could cause a problem as the server is in production.

But creating snapshot was hanging with Veeam not only one only one VM, it was different VM every time.

I created a service request on Veeam, the discovered a logs, and give me an answer that:

Veeam is sending request to esxi but doesn't received and answer. They say that Veeam is only requester in this case, but Vmware is operating with creating/deleting snapshots and probably there is more helpfull info in Vmware logs.

[nhilamoud]

Hello,

is there any account dedicate for veeam already added on PF Vmware with the right previledg ? have you test this acount directly on your Vcenter ? 

Have you ever done a test directly on your vcenter? Have you been able to check the flow openings in both directions (PF VMware <=> Veeam backup) ?

Best regards

Mahmoud Enhila

[BohdanKotelyak]

Hi.

I'm using root account for veeam to reach ESXi.

I'm not using vcenter. Only standalone Esxi.

What do you mean flow opening? How can i check it?

[nhilamoud]

I think it will be good if you create a new account locally on vcenter with the right privilege to use it for Veeam.

Also, have you checked if the ESX is correctly added?

For the flow, it's about the communication between your ESX and your Veeam backup server, it's should already done.

Best regards

Mahmoud ENHILA

[BohdanKotelyak]

Yes.

It's added correctly. If i restart ESXI host - first time backups goes fine (from time to time). Thats why Esxi is connected to veeam correctly.

[nhilamoud]

is there any error message from side Veeam console ?

[PatrickDLong]

@BohdanKotelyak  Just a hunch, but does this host boot from USB or SD/microSD device?  ssh to the host and see if you can list the filesystem with   'ls -n'    Does the command fail to complete?  Does anything show up in red?  If yes then issue:  cat /var/log/vmkernel.log and look for entries like vmhba32 timed out or status in doubt.

[BohdanKotelyak]

Yes. I'm running on USB drive.

[PatrickDLong]

@BohdanKotelyak  From your screenshot it appears your filesystem is reachable because it appears in blue.  If the links to bootbank and altbootbank appeared in red it would mean that your filesystem is not reachable, likely in an APD situation to the boot device.  There are many issues with ESXi 7 (particularly U2) running on USB-based boot media, including motherboard-mounted SD-cards, etc.  Most of the issues are related to the host losing connectivity to the boot device filesystem intermittently or permanently, and more serious cases have resulted in boot device corruption.  If your host cannot see bootbank and altbootbank, you can likely recover from this without rebooting by following @LucianoPatrão 's excellent blog post here:  https://www.provirtualzone.com/vsphere-7-update-2-loses-connection-with-sd-cards-workaround/ 

I would recommend that you evacuate one of these hosts, then  make sure you have redirected scratch to a persistent location like local disk or SAN volume, and relocate your vmTools bits either to ramdisk or to other location like shared storage.  These actions will reduce the amount of I/O going to your boot device and allow for greater stability.  Then move some vms back the the host and retry your backup operations - let us know if you still have issues.

Again, this may not be the cause of your issue, but if your scratch (and vmtools) is still pointing to USB you are effectively overwhelming the I/O capabilities of the device and you will have a myriad of issues.

[IRIX201110141]

I have a customer which face the same problem.... update a Dell R540 (With Dual SD) from 6.7 to 7.0u2 latest together with Veeam. The problem is not related to creating a Snapshot...  ESXi stops answering to requests after some time and starts struggling. I can see "too may requests" in hostd.log.

I cant list the filesystems anymore, problem to access the logs. The Hostclient becomes unresponsable and the console doesnt work anymore(Customer have vCenter too). Also stopping a VM can takes a looonnnnnnng time.

After a ESXi restart it works for a couple of days and than problmes comes back.

Regards,
Joerg

PS: We always redirect scratch partitions and syslog to local VMFS (its a single host setup)

[PatrickDLong]

@IRIX201110141   1-Regain control of hosts by following steps in this blog post to rescan storage:  https://www.provirtualzone.com/vsphere-7-update-2-loses-connection-with-sd-cards-workaround/ 

2- Remediate hosts by minimizing I/O to the low-endurance SD boot media- move scratch partition (host logging, etc.) off SD boot device onto local high-endurance media or SAN shared storage

3- move vmTools references off of SD boot device by enabling UserVars/vmToolsRamdisk or by redirecting productLocker symlink to shared storage location

4- wait for 7.0u3 patch in July or August

[IRIX201110141]

@PatrickDLong 

Read the excellent blog some minutes ago and i will try to implement the work around.

Regards,
Joerg

[f-neumann]

Same problem here. HPE Standalone Host on Update 2(a). Boot from interal usb-drive.
After a few days suddenly no Veeam Backup (Application Error, could not initiate NFS filestream from datastore), no manual Snapshot Creation on host itself (stuck at 0%). VM status not reported correctly (guest shutdown, vm still shown active).
Logfiles inaccessible using SSH, session keeps hangig when accessing/listing filesystem.

Current workaround: manual shutdown of every single vm, reset host, cold boot.

As this happens every 2-3 weeks we'll try to get along with periodic reboots until Update 3. Hope this will be fixed! Total mess.

[knskc]

is your VM have updated tools and vm compatiblity hardware ?

[MCEdel]

Same problem here on DELL server 740 and  ESXi-7.0U2a,
a "little bit" annoying in production servers...

Backups and replicas work fine for a while and then everythhing stucks.
Stopping the replica/backup job does not solve the problem of course.

Once hanging, every backup/replica get NFC storage connection is unavailable etc etc.
In addition, from vcenter server we cannot access datastore and cannot open remote console/web console.
From vcenter we also cannot reboot/shutdown any VM.

Workaround: manual shutdown/reboot of the ESXI host.

Waiting for U3...

[IRIX201110141]

What are you wating for? The Patch for ESXi is already there and named ESXi-7.0U2c-18426014-standard

Regards,
Joerg

[MCEdel]

Thanks, will try soon. 

Best,

MC