Help - Snapshot Errors following Zero diskspace issue

[fastpud]

Hi,

I have an issue which occured over the weekend. On my ESXi 4.1 hosts all snapshots are saved into single datastore. Over the weekend this datastore filled up, halting 6 VM which had snapshots running on them. The VM were frozen when I can in this morning.

Message from 10.0.0.xx: There is no more
space for virtual disk Server1-000002.vmdk
. You might be able to continue this session by
freeing disk space on the relevant volume, and
clicking Retry. Click Cancel to terminate this
session.
info
24/09/2012 08:31:57
Server1

User

I came in a cleared down a load of ISOs to free up space. I answerved the retry question on the VM to get them back working and all seems fine - data is up to date etc.

I then tried to remove the snapshots. However I can't remove or create any snapshots on these 6 VM. When i try to I get the following errors:

Server1

Cannot complete the
operation because the file or
folder <unspecified filename
> already exists
Domain\Admin
Server1.hiqa.local
24/09/2012 08:53:19
24/09/2012 08:53:19
24/09/2012 08:53:24

Create virtual machine snapshot
Server1
Cannot create a quiesced
snapshot because the create
snapshot operation
exceeded the time limit for
holding off I/O in the frozen
virtual machine.
Domain\admin
Server1.Domain.local
23/09/2012 00:09:49
23/09/2012 00:09:49
23/09/2012 00:10:29

I don't know where to go from here to get them back into a stable state. Don't want to do anything in case I make it worse.

Any help would be apprepicated.

Regards,

Fastpud

[a_p_]

In this case I'd recommend you contact VMware support.

If you like, post (attach) the latest vmware.log file for one of the VM's on which this message appeared to see whether this shows what's wrong. Maybe it's only a synchronization issue between the hosts and vCenter Server which can be fixed with either restarting the Management Agents on the hosts and/or the vCenter Server service. However, be careful with restarting if you use other applications like VMware View which rely on vCenter Server.

André

[fastpud]

Andre,

Here are the last 3 vmware.log files as requested. The disk filled up approx 9:00pm on Sat 22nd, one VM's snapshot file exploded in size due to rogue backups.

Regards & Thanks.

Fastpud

[a_p_]

To me this looks like a delta file has been created without the associated descriptor .vmdk file. To see whether it is save to e.g. delete possibly orphaned files however, I need more details. Please provide a list of files in the VM's folder. Since the Datastore Browser does not show all files, you need to get the list from the command line (e.g. SSH) running ls -lisa. In addition to this list, please post (attach) the current .vmx file.

André

[fastpud]

Andre,

Here is the listing of the VM's files

/vmfs/volumes/4d4bdbe8-f8ad714b-ddbf-78e7d1e93f4e/HIQASRV2139 # ls -lisa
    836   64 drwxr-xr-x    1 root     root               2520 Sep 24 08:12 .
      4 1024 drwxr-xr-t    1 root     root               3220 Sep 12 08:41 ..
662708356   64 -rw-r--r--    1 root     root                 13 Sep 24 07:54 HIQASRV2139-aux.xml
490741892 4096 -rw-------    1 root     root            3277312 Sep 14 13:34 HIQASRV2139-ctk.vmdk
532682820   64 -rw-r--r--    1 root     root                 37 Jun 20 15:41 HIQASRV2139-d176768e.hlog
8389444 30393344 -rw-------    1 root     root        53687091200 Sep 18 10:49 HIQASRV2139-flat.vmdk
704647684   64 -rw-------    1 root     root               8684 Sep 24 07:45 HIQASRV2139.nvram
671093252   64 -rw-------    1 root     root                611 May  5  2011 HIQASRV2139.vmdk
25166660   64 -rw-r--r--    1 root     root                 78 Mar  7  2011 HIQASRV2139.vmsd
16778052   64 -rwxr-xr-x    1 root     root               3768 Sep 24 08:18 HIQASRV2139.vmx
20972356   64 -rw-r--r--    1 root     root               1575 Sep 24 08:15 HIQASRV2139.vmxf
146805252 1024 -rw-r--r--    1 root     root             190358 Dec  4  2011 vmware-24.log
176165380 1024 -rw-r--r--    1 root     root             117164 Dec 11  2011 vmware-25.log
75498308 2048 -rw-r--r--    1 root     root            1198917 Jun 17 15:40 vmware-26.log
281022980 1024 -rw-r--r--    1 root     root              72277 Jun 20 15:35 vmware-27.log
247472260 1024 -rw-r--r--    1 root     root             906351 Sep 18 10:51 vmware-28.log
398463492 1024 -rw-r--r--    1 root     root             164299 Sep 24 08:03 vmware-29.log
411046404 1024 -rw-r--r--    1 root     root             156742 Sep 24 13:32 vmware.log

Here is the listing of the same VM's Snapshot location

/vmfs/volumes/4ea6b3f1-8a32878b-89bd-78e7d1e93f4c/VM-Snapshots/HIQASRV2139 # ls -lisa
150996548   64 drwxr-xr-x    1 root     root               1820 Sep 24 14:11 .
   1604   64 drwxr-xr-x    1 root     root               5180 Jun 22 10:16 ..
482353284 4096 -rw-------    1 root     root            3277312 Sep 18 10:49 HIQASRV2139-000001-ctk.vmdk
473964676 11653120 -rw-------    1 root     root        11928707072 Sep 24 13:32 HIQASRV2139-000001-delta.vmdk
478158980   64 -rw-------    1 root     root                458 Sep 18 10:49 HIQASRV2139-000001.vmdk
729817220    0 -rw-------    1 root     root             106496 Sep 22 23:10 HIQASRV2139-000002-delta.vmdk
725622916   64 -rw-------    1 root     root              30565 Sep 22 23:10 HIQASRV2139-Snapshot93.vmsn
788537476   64 -rw-------    1 root     root              30464 Sep 24 07:54 HIQASRV2139-Snapshot94.vmsn
792731780   64 -rw-------    1 root     root              30564 Sep 24 08:05 HIQASRV2139-Snapshot95.vmsn
796926084   64 -rw-------    1 root     root              30564 Sep 24 08:10 HIQASRV2139-Snapshot96.vmsn
805314692   64 -rw-------    1 root     root              30565 Sep 24 08:48 HIQASRV2139-Snapshot97.vmsn
809508996   64 -rw-------    1 root     root              30565 Sep 24 09:37 HIQASRV2139-Snapshot98.vmsn
826286212   64 -rw-------    1 root     root              30564 Sep 24 14:11 HIQASRV2139-Snapshot99.vmsn

I think the snapshot is active as all changes made post snaphot are still on the server.

Thanks

Fastpud

[a_p_ · Accepted solution]

To me this looks like "HIQASRV2139-000002-delta.vmdk" was created, but never used due to the disk space issues. For the virtual disk to be usable the has to exist a descriptor file "HIQASRV2139-000002.vmdk" too. What you may do is to shut down the VM (I'm pretty sure this is not really necessary, but for safety reasons I'd recommend it), then move the "HIQASRV2139-000002-delta.vmdk" to a sub folder or rename it and power on the VM again. If the VM powers on without issues you may delete this file.

André

[fastpud]

Andre,

You were spot on.  Once I deleted this file I was able to remove all the snapshots and server is as I expected. This snapshot would of been created by a Backup exec job which runs over the weekend. I will have to review all other VMs for this issue.

Many thanks, You are a life saver.

Fastpud.