esxi5 VM won't start after running out of data drive space

[captjwh]

Running esxi5.0.1 with the latest patches.  VM is running windows 2008 Server standard, C: = 80GB, D:=200GB Both THIN Provisioned (why is water under the bridge).

Anyway, there were two snapshots on this VM.  While putting data on the 😧 drive, the drive was ran out of space and the VM froze and displayed a Cancel or Retry dialogue.  I tried Retry, but no success.  So while the machine was down, i deleted the snapshots.  that was successful in the GUI, but when i look at the datastore, the snaps are still there.  I tried to start the VM again, but i keep getting "A general system error occurred.  The virtual machine could not start".

The c: drive still has space and i could likely attach it to a new VM and get some critical data off, but i'm really concerned about getting data off the 😧 drive with the snapshot files still out on the datastore if i try to attach it to a functioning VM.  Here is a pic of the datastore for the VM.

Any good ideas here?  i just need the data at this point, but i would be very happy to recover the VM.

thanks!

Jason

[RParker]

If you are *SURE* those snapshots are deleted (committed) from ESX you can delete them manually, by deleting ONLY the snapshot files don't touch the original *.VMDK files or you will delete your VM.

The datastore probably isn't able to allocate space for the VM paging file.  You can get around this by making ALL of the RAM in that one VM "reserved", that way ESX will not try to create a paging file.  Also turn off logging for that VM, it should start now...

IF your VM is thin provisioned, and there is ZERO space left on the datastore it will probably freeze again..

[MKguy]

Did your datastore actually run out of space or just your VM's D:\ drive inside the guest?

You're missing the datastore browser pic. Anyways, in addition to this, can you also show what it looks like on the raw filesystem via ls -l /vmfs/volumes/datastore/vmname/?

The datastore browser usually wants to be clever by bunching the flat-vmdk and vmdk descriptor files together etc, obfuscating the real state at times.

Can you also post the vmware.log file in this folder or excerpts from when you tried to start it?

[captjwh]

Thanks for the quick reply.  Snapshot manager shows no Snaps.  So if that is a valid source of information, then we are ok there.  The Datastore itself has 112GB open space, all good there.  Good idea on the RAM, i'll try that.

I see your second post.  I'll try to c/p in the pic again..actually i just attach.

[captjwh]

I attached the log and the ls -l.  I'm going to try the RAM/log trick now.  OK to whack those snaps you think?  I suspect it is these files?

-rw-------    1 root     root         3540406272 Jan 19 06:26                File-Server-1_1-000001-delta.vmdk
-rw-------    1 root     root                334 Jan 19 04:38                     File-Server-1_1-000001.vmdk
-rw-------    1 root     root        84138151936 Jan 21 05:02                File-Server-1_1-000002-delta.vmdk
-rw-------    1 root     root                341 Jan 19 06:32                     File-Server-1_1-000002.vmdk

thanks.

Let me know if you want more of the log file.

[captjwh]

No Joy on the startup.  I'm getting the same General error and a notfication on the summary screen the the Virtual Machine Disks need cosolidation.

mem 4GB/ reservation 4GB.  logging box unchecked.

[MKguy]

You may want to try cloning the whole VM to another VMFS volume via vCenter or just the disks with vmkfstools -i File-Server-1_1-000002.vmdk /vmfs/volumes/NewDatastore/ -d zeroedthick (see http://kb.vmware.com/kb/1028042).

This will consolidate the snapshots on the destination too and you'd have a backup at least.

[RParker]

I think what happened is you are basically screwed..

Your snapshots are there, but VMware didn't commit the changes, but it removed them from the VM window.   You need to find other files elsewhere on this datastore you can delete.

Or as another poster suggested, clone this VM as is then delete original... I think these snapshots are still valid, the commit didn't have room to merge the files with original VMDK.. I hope you have good backups.. your VM may very well be hosed...

[captjwh]

Thanks guys.  I found a post about consolidation snaps and found that my vmdk CID's are out of whack with the parent.  i'm backing up what i got then i'll try their suggestion to realign.  i'll let you know what happens.  thanks for the quick responses.  i'll continue to monitor the thread if anyone else has any ideas.

Another idea....

Can i simply remove the 😧 drive and boot the VM again?  i realize windows will be ticked, but at least i'm ahead of the game.

Secondly, assuming i can get this CID alignment thing works, is there any way to mount the hard drive to another VM to have a chance at removing some data?

thanks.

[RParker]

I don't think removing 😧 will make a difference, the datastore is pretty much the entire problem.. no free space = disaster for powering on and currently running VM's

You can always attach a VMDK to any other VM at any time... since the VM is off just attach it, and you an see the data as needed, that should work just fine.

[captjwh]

The datastore has plenty of space.  so i'm going to backup everything and then see if I can attach to anohter VM.  Also, how can i be 100% sure that when i deleted my snapshots that they are no longer in use?

Also, if i attach to anothe VM, will this VM crash too because that hd is full?

[RParker]

CaptJWH wrote:

The datastore has plenty of space.  so i'm going to backup everything and then see if I can attach to anohter VM.  Also, how can i be 100% sure that when i deleted my snapshots that they are no longer in use?

Also, if i attach to anothe VM, will this VM crash too because that hd is full?

100% certain, identify date\time file stamps.  If the VM will not boot, it's not getting that far.. you can always rename a snapshot boot the VM, if the VM does not boot after the snapshot change, but did before, there is your answer.  100%.  If it does boot, now you can safely delete the snapshots.

Windows OS will not crash, only the boot or place where paging is stored, that will cause OS to freeze.. but not usually crash.  So attaching a full vmdk to a running VM should not cause that running VM to crash, the VM will see the newly added disk as simply data.. it has no reference or connection to anything on that drive.

[captjwh]

cool.  I really appreciate your input.  i'm backing up everything now.  Will get back to work in a bit.  Thanks