Neither snapshot manager nor "vim-cmd vmsvc/snapshot.get" see snapshots

[racom]

We run one VM with 4 virtual disks on reserved ESXi 5.5 server. After the VDP backup job failed I've found snapshot files *-0000xx-delta.vmdk as well as corresponding *.vmsd file. There were no snapshot entries in the Snapshot Manager so I tried to consolidate snapshots following http://kb.vmware.com/kb/1002310

But it looks like ESXi host doesn't know about snapshots:

~ # vim-cmd vmsvc/getallvms

Vmid    Name                                File                               Guest OS           Version   Annotation

2      VMname  [datastore] VMname/VMname.vmx   oracleLinux64Guest   vmx-10

~ # vim-cmd vmsvc/snapshot.get 2

Get Snapshot:

~ #

If I try to create a new snapshot it failed almost immediately but new *-0000x+1-delta.vmdk files are created and no error message is in vmware.log.

~ # vim-cmd vmsvc/snapshot.create 2 test test_snapshot 0 0

Create Snapshot:

Create snapshot failed

If I include memory to snapshot creating it runs quite a long time but failed the same way but I can see "SnapshotVMXTakeSnapshotComplete: Done with snapshot 'test': 325" in the end of vmware.log. In hostd.log I can see some warnings:

2015-02-26T16:43:02.369Z [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] Snapshot tree refresh ignored because storage is not accessible.

2015-02-26T16:43:02.371Z [FFE40B70 warning 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] PopulateCache failed: _diskAccess : true, _storageAccessible : false

2015-02-26T16:43:02.371Z [FFE40B70 warning 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] PopulateCache failed: _diskAccess : true, _storageAccessible : false

...  dtto ...

2015-02-26T16:43:02.381Z [FFE40B70 verbose 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] Time to gather config: 11 (msecs)

2015-02-26T16:43:02.381Z [FFE40B70 warning 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] UpdateStorageAccessibilityStatusInt: The datastore 53a183b5-4367b267-c93d-180373fa1be8 is not accessible

2015-02-26T16:43:02.381Z [FFE40B70 verbose 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] CfgUpdate refresh all core files.

2015-02-26T16:43:02.381Z [FFE40B70 warning 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] CannotRetrieveCorefiles: VM storage is inaccessible.

2015-02-26T16:43:02.382Z [FFE40B70 verbose 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] Time to gather snapshot file layout: 0 (msecs)

2015-02-26T16:43:02.382Z [FFE40B70 warning 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] CannotRetrieveCorefiles: VM storage is inaccessible.

2015-02-26T16:43:02.382Z [FFE40B70 warning 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] Snapshot create succeeded, but UID 325 not in list.

2015-02-26T16:43:02.382Z [FFE40B70 warning 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] Failed operation

2015-02-26T16:43:02.383Z [FFE40B70 info 'Vmsvc.vm:/vmfs/volumes/53a84235-a1905d30-5fe4-ecf4bbc24bf0/VMname/VMname.vmx'] State Transition (VM_STATE_CREATE_SNAPSHOT -> VM_STATE_ON)

Datastore couldn't be unaccessible I was runnig commnad in it and was able to list it. I'm can't reboot VM nor host in the nearest future. Any idea how to commit snapshots, please?

[roneng]

take a look at that also

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=201707...

in my case, sometimes i just had to migrate the vm to fix this

good luck

[GreatWhiteTec]

VDP and VDPA have many of the EMC Avamar features. I don;t use VDP but I do use Avamar's VADP backups and had the same issue. Long story short. The Avamar job had the snapshot held hostage and the Host/VM had no trace of a snapshot. We had to remove the snapshot/job from Avamar and that fixed the issue. The quick test was just to power off the Avamar proxy and that made the VM happy again. This could be the same or similar scenario in your case.

[racom]

Thank for your tip. I've completely deleted previously disabled VDP backup job for this VM but it didn't help. I rebooted VPD Appliance then but with the same result. I'll get a question in Backup & Recovery forum.

[racom]

Thanks for your answer and link but files don't look like locked from the perspective of ESXi host. It'll be probably VDP broken backup problem. I can't migrate 'cause it's a dedicated server for this VM and vMotion is disabled.

[HHRS]

Hi Racom

Currently I test a new Tool called Snapwatcher (is still beta).

Is a Little tool to find snapshots and are they corrupt (inconsistent snapshots) this tool can fix it.

it told you in case of a backup disk hot-add behavior which VM has mapped the vmdk...

I think it's very interessting for troubleshooting in large environments.

here the link to the public beta:

http://try.opvizor.com/snapwatcher

[racom]

Hi HHRS,

thank you for tip. I've downloaded an installed snapwatcher but even it doesn't see this snapshots. As dvdmorera said the Host/VM/vCenter probably knows nothing about snapshots made by Avamar/VDP.

It looks to work fine for common snapshots.

[opvizor]

Thx Racom for checking out Snapwatcher. There is most of the time a way to detect snapshots if the VM runs on that. No matter if vCenter knows about it or not. The only exception is, that no snapshot is made at the VMware level but only on the storage level.

I don't think that Avamar is avoiding VMware snapshots.

That said, of course we like to support you on that matter. I'll contact you seperately to receive your log files and vmx files.

Dennis

www.opvizor.com

[racom · Accepted solution]

Reboot of the host solved the problem in the end. Snapshots became visible and I can remove them. I have no idea what caused this situation.

Thanks to opvizor for support and kind assistance.