Issue with snapshots not removing

[jaken]

This started when trying to power on a VM after a power failure on my ESXI host. Anyways one of my VM's will not power back one, give me the error 'Cannot  open the disk  '/vmfs/volumes/4a365b5d-eceda1-19-439b-000cfc0086f3/examplevm/examplevm-000001.vmdk'  or one of the snapshot disks it depends on.'

So I've tried the following:

http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1007969

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=100231...

Still no luck, I currenly have to delta files or snapshots, and for some reason I cannot get my VM to revert back to these. Not sure what to do know, anyone have any ideas?

Thanks.

[jaken]

Sorry, Yea I meant that I changed the file size within the vsphere client and NOT the actual vmdk files.

[continuum]

ok - then we go on with 101 Gb size from now on.
We can not revert the size change of the basedisk easily.
So we adjust the snapshots.

So please stop everything like copy for now and delete the 3 small vmdks - replace them with the ones I attached and try the

vmkfstool -i "ManagementServer-000002.vmdk" "mgmt-server.vmdk"

again.
Sorry - for the confusion

[jaken]

I copied over the files, still get the same error. Below is ouput and also the list of all files in the directory.

# vmkfstools -i "ManagementServer-000002.vmdk" "Mgmt-server.vmdk"

Destination disk format: VMFS zeroedthick

Failed to open 'ManagementServer-000002.vmdk': Input/output error (327689).

/vmfs/volumes/4daef30d-402d8790-fcb7-0019d1ddfc33/ManagementServer # ls -lah

drwxr-xr-x    1 root     root         2.6k Aug 23 16:18 .

drwxr-xr-t    1 root     root         1.8k Aug 23 16:36 ..

-rw-------    1 root     root         2.7G Jun  8 20:40 ManagementServer-000001-delta.vmdk

-rw-------    1 root     root          258 Aug 24  2011 ManagementServer-000001.vmdk

-rw-------    1 root     root        23.1G Aug 22 09:27 ManagementServer-000002-delta.vmdk

-rw-------    1 root     root          265 Aug 24  2011 ManagementServer-000002.vmdk

-rw-r--r--    1 root     root           13 Aug 23 12:21 ManagementServer-aux.xml

-rw-------    1 root     root       101.0G Aug 22 11:55 ManagementServer-flat.vmdk

-rw-------    1 root     root         8.5k Aug 23 09:09 ManagementServer.nvram

-rw-------    1 root     root          341 Aug 24  2011 ManagementServer.vmdk

-rwx------    1 root     root         2.9k Aug 23  2011 ManagementServer.vmx

-rw-------    1 root     root         1.5k Aug 23 13:24 ManagementServer.vmxf

-rw-r--r--    1 root     root       194.7k Aug 23 09:04 vmware-42.log

-rw-r--r--    1 root     root       182.9k Aug 23 09:09 vmware-43.log

-rw-r--r--    1 root     root        31.1k Aug 23 09:27 vmware-44.log

-rw-r--r--    1 root     root        33.0k Aug 23 12:19 vmware-45.log

-rw-r--r--    1 root     root        33.0k Aug 23 12:20 vmware-46.log

-rw-r--r--    1 root     root        33.0k Aug 23 12:22 vmware-47.log

-rw-r--r--    1 root     root        33.0k Aug 23 13:26 vmware.log

[continuum]

not good
so let me summarize for now
we tried to clone each snapshot in to a new vmdk using clean edited descriptor-vmdks from me - in both sizes 100Gb and 101 Gb
all clone attempts failed

in my experience this most probably means: the snapshots were not written completely to disk while your ESXi crashed.
This means they have holes.
ESXi does not have tools to fix this.
VMware Workstation has tools to fix this holes - if and only if the damage is not too serious.

It is midnight here and I have to rise early. So we call it a day for now.
Tomorrow I am on the road and can not look into this.

We can arrange something for thursday.

In the mean time you could do a few things.

1. try to copy the directory with datastorebrowser to a  new location on the ESXi
- this maybe useful if the problem is with the VMFS filesystem itself

2. find a machine with XP 32 or 2003 32 and download the VM directory to the Windows machine.
A USB-disk with NTFS would be good enough to store the directory.
Also download VMwaorkstation 7.1.4 and install it.

3. if both fails download a Ubuntu 64bit LiveCD  from here http://www.ubuntu.com/download/ubuntu/download select the 64bit version

4. install skype and teamviewer on the 32 bit Windows machine

Please decide wether this VM is important enough  to go on ... we may need several hours to try the steps I have in mind

[continuum]

in case Andre has more ideas - trust him - you are in good hands.

I am running out of options - other than the ones that require that I work on the case myself

[jaken]

Well in trying to copy all of the files to my local workstation I run into errors. This only occurs for the flat file and the 000002 delta file. I've tried using the datastore browser, WinSCP, and PSCP. For some reason they just hang after a while, thinking it might be my pc I tried another one, still the same issue. The files timeout when transferring them and cant figure out why.

Looks like I'm at the point to just scrap the idea on trying to recover the server and just start over.

Unless you or anyone else has any ideas.

Thanks.

[continuum]

Hi
bad news - but somehow I expected this outcome - thats why I told you to download a Ubuntu LiveCD

with that LiveCD you can boot the ESXi with the LiveCD, attach a USB-drive and mount the VMFS-volumes.
Often vmdks that can not be copied using other tools or procedures can still be copied with the LiveCD.

To be able to do this you need:

• the UbuntuLiveCD
• the ESXi should have one network card that can get direct access to the internet - DHCP would be nice
• a USB-external drive large enough to store the vmdksyou want to recover

You need internet-access as the Ubuntu-LiveCDs dont have the package vmfs-tools on board.
So you need to boot into Ubuntu Trial-mode - then establish a working internet connection.
Then you start Synaptic package manager and allow Ubuntu to use all preset sources. Needs two activated checkboxes in the source-selection screen that are now unchecked.
Then you do an upgrade of the system - when that is done you can install "vmfs-tools" with Synaptic.

vmfs-tools are a command line tool that is quite easy to use.
See this site for an explanation

http://planetvm.net/blog/?p=1592

If necessary I can talk you through the procedure on the phone or skype - but it is not too hard.

Well - this is a procedure for the desperate - even if you copy the files you still cant be sure they are readable or can be repaired.

I only use this for really IMPORTANT data ...

[jaken]

Hi Continuum,

As I would like to try and work the problem out and as much as I appriciate all of the help that you've provide I may just call it quits. Part of me want to try and see if its possible to get this VM working again but the other part just wants to move on. I've rebuilt a new VM and have got it setup to basically mimic the old one and is functioning properly. So if you're ok with not going forward, I'm fine with that.

Again, thanks for your help.

Jake

[continuum]

NO - that is not ok.
I already canceled my weekend trip to Paris just to fix your VM. You cant give up now !!!

Just kidding ...
The case has aroused my curiousity and I would have liked to see deeper into it.
But then I would do the same - if this VM has no mission-critical data that MUST be restored it is not really worth any more effort.

Ulli

[jaken]

Haha, yea part of me wants to try an get it fixed but its just not necessary. I'm just glad it happend to a non mission critical machine.

Thanks for the help Ulli!

Jake