VMware Cloud Community
alexp_789
Contributor
Contributor

Unable to take Snapshots - Unable to save snapshot file

Hi All,

Well i don't know why, but snapshots have stopped working on 'some' of my virtual machines.... The error i get in VC is:

A general system error occured: Unable to save snapshot file

Now the weird bit, if i reboot the virtual machine the problem persists, however if i power the machine off, and then re-start it, it magically works again!?!?!?

I have two ESX servers, and i've tried migrating the machines between them and the problem occurs on both, i'm using a NetApp filer and iSCSI for storage.

I'm posting this 1) so i hopefully can fix this issue, and 2) so other people know that simply powering the virtual machines on and off will fix the problem (if you can call it a fix!

Thanks in advance all!!

Alex

Reply
0 Kudos
31 Replies
CHogan
VMware Employee
VMware Employee

Could this be a disk space issue?

Do you have enough space available to save the snapshots?

http://cormachogan.com
Reply
0 Kudos
kix1979
Immortal
Immortal

Like CHogan said, check your disk space. Snapshots add a lot of diskspace REALLY fast. On a side note, I would wait to use snapshots for the first patch. There are a lot of ways you can get into trouble with them really fast. I have had to recreate vmx files and delete vmdk snaps because of issues running CLI or VI Client commands.

Thomas H. Bryant III
Reply
0 Kudos
sebek
Enthusiast
Enthusiast

I had got same problem. As I observed, but it's my home opinion, snapshots depends on cpu load. I don't know if it helps, but in my environment I solved it by vmotioning vm's to another esx box with less cpu load.

I migrated snapshotted vms back to its home esx and tried to increase sc resources (cpu shares). It helped a little, too.

Reply
0 Kudos
JesseG
Contributor
Contributor

I am having this exact same issue. Server load and storage space is not a factor. It looks like a bug to me.

Jesse

Reply
0 Kudos
CHogan
VMware Employee
VMware Employee

Anything of interest in the vmware.log of the VM that is being snapshot'ed?

http://cormachogan.com
Reply
0 Kudos
alexp_789
Contributor
Contributor

Very sorry about the delay all, hope you've not lost interest! Smiley Wink

Here's a snippit of the log, the load on the server isn't excessive, and the is 300gb + disc space yet, i'm using iSCSI, shared between two servers, and when the snapshots fail on one, they still work on the other, so i'm discounting a iSCSI issue? (well the server anyway!)

Thanks in advance!

Aug 01 08:16:19.806: vcpu-0| GuestRpc: Channel 6, registration number 1, guest application toolbox-dnd.

Aug 01 08:16:19.806: vcpu-0| DISKUTIL: scsi0:0 : toolsVersion = 7172

Aug 01 08:17:21.092: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : open successful (21) size = 9437184000, hd = -1. Type 3

Aug 01 08:17:21.093: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : closed.

Aug 01 08:17:21.203: vcpu-0| Sync monModules(1).

Aug 01 08:17:21.203: vcpu-1| Sync monModules(1).

Aug 01 08:17:21.203: vcpu-0| Done Sync monModules(1).

Aug 01 08:17:21.205: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : closed.

Aug 01 08:17:21.207: vcpu-0| Cpt monModules(2).

Aug 01 08:17:21.207: vcpu-0| Done Cpt monModules(2).

Aug 01 08:17:21.207: vcpu-1| Cpt monModules(2).

Aug 01 08:17:21.207: vmx| MigrateOpen: migration ID not set

Aug 01 08:17:21.208: vmx| DUMPER: failed to open checkpoint file: No such file or directory

Aug 01 08:17:21.226: vcpu-0| Sync monModules(5).

Aug 01 08:17:21.227: vcpu-0| Done Sync monModules(5).

Aug 01 08:17:21.227: vcpu-1| Sync monModules(5).

Aug 01 08:17:21.228: vmx| DISK: OPEN scsi0:0 '/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk' persistent R\[(null)]

Aug 01 08:17:21.339: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : open successful (10) size = 9437184000, hd = 8246. Type 3

Aug 01 08:17:21.339: vmx| DISKLIB-DSCPTR: Opened : "kayaking-flat.vmdk" (0xa)

Aug 01 08:17:21.343: vmx| DISKLIB-LINK : Opened '/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk' (0xa): vmfs, 18432000 sectors / 9000 Mb.

Aug 01 08:17:21.343: vmx| DISKLIB-LIB : Opened "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk" (flags 0xa). 0x85f2d50

Aug 01 08:17:21.427: vmx| DISK: OPEN '/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk' Geo (1147/255/63) BIOS Geo (0/0/0) freeSpace=376544Mb, DDB.adapterType=lsilogic

Aug 01 08:17:21.427: vmx| DumpDiskInfo: scsi0:0 createType=11, capacity = 18432000, numLinks = 1

Aug 01 08:17:21.427: vmx| SCSIDiskESXPopulateVDevDesc: Using FS backend

Aug 01 08:17:21.859: vcpu-0| HBACommon: First write on scsi0:0.fileName/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk

Aug 01 08:17:21.864: vcpu-0| DISKLIB-CHAIN : UpdateContentID: old = 0x4f2a773c, new = 0x7f40b2a6

Aug 01 08:19:58.542: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : open successful (21) size = 9437184000, hd = -1. Type 3

Aug 01 08:19:58.543: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : closed.

Aug 01 08:19:58.624: vcpu-0| Sync monModules(1).

Aug 01 08:19:58.624: vcpu-0| Done Sync monModules(1).

Aug 01 08:19:58.624: vcpu-1| Sync monModules(1).

Aug 01 08:19:58.629: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : closed.

Aug 01 08:19:58.632: vcpu-0| Cpt monModules(2).

Aug 01 08:19:58.632: vcpu-1| Cpt monModules(2).

Aug 01 08:19:58.632: vcpu-0| Done Cpt monModules(2).

Aug 01 08:19:58.632: vmx| MigrateOpen: migration ID not set

Aug 01 08:19:58.632: vmx| DUMPER: failed to open checkpoint file: No such file or directory

Aug 01 08:19:58.640: vcpu-1| Sync monModules(5).

Aug 01 08:19:58.640: vcpu-0| Sync monModules(5).

Aug 01 08:19:58.641: vcpu-0| Done Sync monModules(5).

Aug 01 08:19:58.641: vmx| DISK: OPEN scsi0:0 '/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk' persistent R\[(null)]

Aug 01 08:19:58.646: vmx| DISKLIB-VMFS : "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking-flat.vmdk" : open successful (10) size = 9437184000, hd = 10296. Type 3

Aug 01 08:19:58.646: vmx| DISKLIB-DSCPTR: Opened : "kayaking-flat.vmdk" (0xa)

Aug 01 08:19:58.647: vmx| DISKLIB-LINK : Opened '/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk' (0xa): vmfs, 18432000 sectors / 9000 Mb.

Aug 01 08:19:58.647: vmx| DISKLIB-LIB : Opened "/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk" (flags 0xa). 0x85e9ce8

Aug 01 08:19:58.717: vmx| DISK: OPEN '/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk' Geo (1147/255/63) BIOS Geo (0/0/0) freeSpace=376544Mb, DDB.adapterType=lsilogic

Aug 01 08:19:58.717: vmx| DumpDiskInfo: scsi0:0 createType=11, capacity = 18432000, numLinks = 1

Aug 01 08:19:58.717: vmx| SCSIDiskESXPopulateVDevDesc: Using FS backend

Aug 01 08:19:58.787: vcpu-1| HBACommon: First write on scsi0:0.fileName/vmfs/volumes/44a2944d-9357a20b-7ddd-00132120038e/kayaking/kayaking.vmdk

Aug 01 08:19:58.793: vcpu-1| DISKLIB-CHAIN : UpdateContentID: old = 0x7f40b2a6, new = 0x631dcaa7

Message was edited by:

alexp_789

Reason: Found better section of log

Reply
0 Kudos
groundLoop
Contributor
Contributor

We found an identical error message in our vmware.log file, after attempting to snapshot a running VM:

Aug 09 10:43:42.265: vmx| DISKLIB-VMFS : "/vmfs/volumes/44cf4d5a-8b70b4f0-b6b0-000423d09f7a/vm_te

st1/vm_test1-flat.vmdk" : closed.

Aug 09 10:43:42.266: vcpu-0| Cpt monModules(2).

Aug 09 10:43:42.266: vcpu-0| Done Cpt monModules(2).

Aug 09 10:43:42.266: vmx| MigrateOpen: migration ID not set

Aug 09 10:43:42.266: vmx| DUMPER: failed to open checkpoint file: No such file or directory[/b]

Aug 09 10:43:42.269: vcpu-0| Sync monModules(5).

Aug 09 10:43:42.269: vcpu-0| Done Sync monModules(5).

Aug 09 10:43:42.269: vmx| DISK: OPEN scsi0:0 '/vmfs/volumes/44cf4d5a-8b70b4f0-b6b0-000423d09f7a/v

m_test1/vm_test1.vmdk' persistent R\[(null)]

We are running into a similar problem. Any progress on your end?

Some observations:

1. Snapshots work with a powered off VM.

2. After powering the VM back on, the snapshot error went away....

Good luck. I have some opinions on this behavior, but I'll have to restrain myself...

-gL

Reply
0 Kudos
garthk
Contributor
Contributor

I'm getting the same, with ESX 3.0 accessing the datastore over FCP.

Reply
0 Kudos
whitehat
Contributor
Contributor

We discovered that snapshots of VMs created from ISO/original install (and not Templates) require the following steps for snapshots with virtual memory to be taken:

1. Take a snapshot without the memory box checked.

2. Revert to the snapshot on the VM

3. Take a snapshot \*with* the memory box checked.

After that, snapshots have no problem. This is confirmed with VI3.0. VMs created from Templates seem to have no problems with snapshots w/ or w/o memory.

Reply
0 Kudos
haywarn
Enthusiast
Enthusiast

This option works:

1. Take a snapshot without the memory box checked.

2. Revert to the snapshot on the VM

3. Take a snapshot \*with* the memory box checked.

However when you perform this it also shuts down the virtual machine.

So you can also shutdown the virtual machine and power it back on and get the same result as mentioned above.

My main concern with this issue (yes I'm also having the problem with VI3) is that I don't want to have to shutdown production servers to perform a snapshot.

I noticed today that I'm not having this issue with a virtual machine that I built from a template. The template was built by building a virtual machine using a Windows 2003 CD. I just cloned the template to create a second virtual machine.

However I am having this issue on all of the virtual machines that were built using the P2V conversion assistant that is sold by Vmware. Is anyone else having this issue with only P2V'ed virtual machines?

I have opened a case with HP (since they provide our support for Vmware). I will post if we get to the bottom of this issue.

Reply
0 Kudos
s1m0nb
Enthusiast
Enthusiast

Bump....

Any progress on this one? I am suffering from exactly the same issue...

Reply
0 Kudos
pzednik
Contributor
Contributor

VMWare pls. respond to this thread!!!

we have the same problem.

It looks that there is heavy need for a lot of hotfixes!!! (VC, ...)

Reply
0 Kudos
kitcolbert
VMware Employee
VMware Employee

Aug 09 10:43:42.266: vmx| DUMPER: failed to open checkpoint file: No such file or directory[/b]

It would have been nice for us to actually print out the file that the VM is trying to open... Oh well.

Has anyone filed an SR?

Reply
0 Kudos
ErMaC1
Expert
Expert

Yes, I have an SR open and have had one for at least a couple weeks. We first started noticing this problem a few weeks ago; turning off the VM and turning it back on fixes the problem temporarily but it comes back at random intervals.

Reply
0 Kudos
haywarn
Enthusiast
Enthusiast

I've opened a case with HP, they have called me back. Hopefully I'll have an answer by tomorrow.

Reply
0 Kudos
kitcolbert
VMware Employee
VMware Employee

Can you give me the SR number?

Note also that there is a known bug that snapshots break after you VMotion a VM. So if you VMotion a VM, then you can't take any snapshots of it until you power off or suspend the VM. Perhaps you're seeing this?

Reply
0 Kudos
ErMaC1
Expert
Expert

Wow okay so that's something that's not documented anywhere and it would've been probably really nice if that was communicated to your techs, because he and I have been scratching our heads at this for two weeks.

The SR is 274985, my tech is Damian.

Reply
0 Kudos
kitcolbert
VMware Employee
VMware Employee

Yeah, we just found out about it last week. So it's hot off the press.

Reply
0 Kudos
ErMaC1
Expert
Expert

OK I called the tech I was working with on this and let him know about this.

If this is the case - that VMotion breaks snapshotting - then this is an bug big enough to drive a TRUCK through because as far as I know Consolidated Backup requires snapshots to operate, which means that VMotion breaks VCB. That's rediculously stupid.

Reply
0 Kudos