Snapshots fail with "A general system error occurred:"

[timofcourse]

Just suddenly in the last couple days, we cannot take snapshots on a handful of our VMs. Trying to do so gets to about 90% and eventually fails with "A general system error occurred:".

Ive done some searching and every result I found that included this error had something after the "has occurred".

Any thoughts on how to get more info on this or what may be causing this?

Thanks!

[a_p_]

Do these VMs have something in common, i.e. located on the same datastore? Do you have sufficient free disk space on the datastore(s)? Did you already check the VM's vmware.log file (in case the snapshots are created with the VM powered on) to see whether it contains additional information about the issue? Does this issue occur when you create a snapshot with the VM powered on and/or powered off?

André

[timofcourse]

Found a bit more info that might be helpful....

First off to answer your question, this is occurring across 3 different Hosts and datastores.

After a bit more digging, I found these VMs do have the following in common:

1. The Hard Disk shows up as "0 MB" in the settings of all VMs

2. Looking at the file listing of these VMs, there are a ton of snapshot files, even though there are no snapshots according to snapshot manager (see attached file-listing.log as an example).

3. All of these VMs are being backed up by Veeam Backup and Replication.

4. I need to look a little further to confirm, but I believe that of all VMs that are being backed up by Veeam, the ones that are not generating this error when creating snapshots do have snapshots according to snapshot manager.

So my thought is this is something related to Veeam. Before I opened a ticket with them, thought I'd get your take on this. Ive also attached the vmware.log as well.

Thanks!

[a_p_]

According to the lof file the snapshots are created by Veeam Backup, but I don't really understand why the aren't deleted anymore. There are error messages in the log file related to the CD-ROM drive, and although I don't see a relation please check whether the VM has a virtual CD-ROM connected and disconnect it if it's not needed to see whether this helps.

Btw. The Snapshot Manager is just a graphical representation of the VM's .vmsd file contents. If the contents in this files are deleted, but the snapshot isn't, it will not show up anymore in the Snapshot Manger. What I usually do is to check my environments using http://www.robware.net/ which is a really helpful tool.

André

[timofcourse]

Youre correct in that all of these machines do appear to have the CD-ROM connected. I've disconnected them all, but still unable to take a snapshot.

I downloaded RVTools and while those extra snapshot files don't show up under vSnapshot tab, under the vHealth tab I can see every one of them with the message "Possibly a Zombie vmdk file! Please check".

A bit more searching has shown others using Veeam have run into this issue and it looks to have been addressed in their latest Patch 4 according to the following release note:

If the same ESX(i) host is added to Managed Servers both as a part of the vCenter, and as a standalone host, the jobs fail to remove VM snapshots correctly.

Which is exactly I think what happened. Unfortunately applying the patch did not fix the existing snapshots, so Ive opened a ticket with them in hopes to get them cleaned up and this issue resolved.

Thanks for the help and suggestions.

[timofcourse]

So Veeam is telling us we should get VMware support involved and wouldnt you know it, our support agreement just expired and need to go through the quote/req/approval process again to get it renewed which will take about a month.

Anyone have recommendations on how to best resolve this issue of removing these "zombie" snapshots and get snapshots working again?

Thanks as always.

[a_p_]

If you want I can take a look at the files to see whether these zombies are obsolete files which may cause the issue. However, I'm a little bit worried about the screenshot you posted earlier where the virtual disk file is a snapshot file and the disk size is zero!?

Anyway, to start with one of the VM's, please create a list of files in the VM's folder (ls -lisa), compress/zip this file along with the VM's .vmx, .vmsd as well as all the .vmdk header files (the small text files without flat, delta or ctk in their names). A screenshot of the zombies related to this VM from RVTools might also be helpful.

André

[timofcourse]

Attached is a zip with requested screenshot and files for one of the VMs we have this issue with. Let me know if theres any other info I can provide.

Thanks a ton!

[a_p_]

I took a look at all the files and they look ok to me, except for the VM's virtual CD-ROM drive which causes and I/O error. It also looks like snapshots are still created on this VM (the latest one "000029" was created at 6PM today)!?

Anyway, please edit the VM's settings and disconnect the virtual CD-ROM drive, i.e. select "Client-Device" and uncheck "Connected" as well as "Connect at Power On" to see whether this changes anything.

André

[timofcourse]

6:00 UTC is 2:00AM EST, which is when our Veeam backup job ran and it tried to make another snapshot as part of this job which also failed with the same "A general system error occurred: ".

As for the CD-ROM, according to the vSphere client its not currently connected:

[a_p_ · Accepted solution]

>>> As for the CD-ROM, according to the vSphere client its not currently connected:

That's correct, but it seems that there's an issue with the ISO file's location, so please set the CD-ROM drive to "Client Device" just to rule this out as a possible issue.

André

[timofcourse]

Is this possible to do outside the vSphere GUI? Because the Hard disk 1 is set to "0 MD", I cant save any changes and Im afraid to enter in any kind of value for the Hard disk size.

Safe to assume I can just update the vmx directly?

[a_p_]

Updating the .vmx file won't help much as you need to reload it after manually updating it. Anyway, the disk size is not part of the .vmx file, but is read from the .vmdk file(s). I'm pretty sure that changing the CD-ROM settings won't hurt the VM. However, I cannot guarantee it.

André

[timofcourse]

Ok then. Think Im finally all set. Phew.

Because I couldn't change the CD-ROM via the GUI due to the Hard Disk issue, I updated the vmx files directly and replaced all ide1 settings with the following:

ide1:0.present = "true"

ide1:0.clientDevice = "TRUE"

ide1:0.deviceType = "atapi-cdrom"

ide1:0.startConnected = "FALSE"

As you thought, I was then able to take a snapshot without issue. Once I did that, all of the other Veeam Snapshots showed up. What a mess...

I was then able to "Delete All" to clean them up.

Looks like Im back in business!

Thanks so much for the help a.p.!

[a_p_]

Great, thanks for the feedback.

It's always interesting how obviously unrelated settings can cause issues. This is certainly something Veeam and/or VMware should look at.

André

[dariusd]

I have added this thread to our internal bug tracker so that the virtual CDROM folks can take a look.  It looks similar to (but more severe than) an issue that can occur when attempting to eject a CD/DVD while the OS has locked the drive tray.  I can't say I fully understand whether or not it's the same issue, but hopefully the experts can figure it out.

Thanks for your patience, timofcourse, and thanks for being awesome as always, André!

--

Darius

[just_larry]

Had this issue today. VMware 6 patched.

Yesterday I deleted a old data store that had iso's on it. One vm had a iso mounted to the CDrom. Every time a snapshot that VM it went to General Error. Disk size then showed 0 gig. If I migrate the VM to another data store it showed the disk at 60gig after. If I tried to snapshot then error back to zero. I then migrated back to first data store, set CD to client, now all works.

[CENSENAT]

I had the same problem with Veeam Backup. I read thi useful articule and resolved with this steps.

Shutdown the VM. Rename, the .vmx file.

Download an edit the .vmx

resent = "TRUE"
sata0.present = "TRUE"
ide0:0.deviceType =
"cdrom-image"
ide0:0.fileName =
"/vmfs/volumes/4a6a5c3f-523a23be/ISOS/SQL_Svr_Standard_Edtn_2016w_SP1_64Bit_English.ISO"

Change the las line.

ide0:0.fileName = ""

Upload with the original name.

Reboot the vm. Take an snapshot (now it Works). When the snapshots finished that appear all the veeam backups snapshot as described in this article. Delete all snapshots and Veeam backup runs ok. That you.

[Maxhawkgt]

I wanted to update this for others who might stumble upon this thread. I had the same problem with Veeam as well as Synology Active Backup for Business where backups were failing. Both my 6.5U1 and 6.7U1 hosts had VMs with a bunch of extra -xxxxx.VMDK files and snapshots that weren't in use. I also had zero-size hard disk size in the affected VMs as well as ISO images tied to the CDROM.

I downloaded the .vmx files and changed my CDROMs to this (mine are sata0 instead of ide):

sata0.present = "TRUE"

sata0:0.deviceType = "atapi-cdrom"

sata0:0.startConnected = "FALSE"

sata0:0.present = "TRUE"

This fixed the zero size HDD but I couldn't get the VM to boot. I then edited the line which specified the vmdk file and now all is well.

scsi0:0.fileName = "OpenVPN.vmdk" instead of scsi0:0.fileName = "OpenVPN-00012.vmdk"

This appears to have been caused by downtime of the storage pool where the CDROM's ISO images were stored. While the storage pool was down, Veeam and Active Backup made snapshots  which resulted in this issue. In the future I'll pause/disable scheduled tools that create snapshots when the storage pool is down. I'll also try to be better about leaving CDROMs connected with an ISO image.