Critical machine running on snapshots Vcenter 6

[Gysh]

Hello everyone I need assistance in a serious and critical issue for a client.

The client is running Vmware Essential 6 (Do not have a subscription that let us call Vmware for support)

So on the Vcenter 6 host we have some Windows 2012 R2 machines and 1 that is in critical state atm, we learned that the machine has been running on snapshots for about 3 months now.  (the disk has the -0000001.vmdk file name

I tried to save the server by making a backup, then creating a new snapshot, consolidating the snapshots and deleting the snapshot and the machine still runs on -0000001.vmdk

I tried to manually edit the .vmx file to point towards the normal .vmdk, which let me machine boot but with beeing behind by 3 months.

So my questions are

1) How to I restore the machine to his normal state (running the .vmdk) while saving all my data, is it possible to do it without starting a new machine?

2) What causes Vmware to sometimes not reconfigure hard drives correctly to stay on -0000001.vmdk snapshot disk when deleting snapshots, this can cause serious issues.

Thanks

[a_p_]

I tried to manually edit the .vmx file to point towards the normal .vmdk, ...

You should never do this unless you are prepared to loose data due to a guest file system corruption! Were you able to restore the snapshot chain (not sure whether this is related to your first question)?

Anyway, a 3 months old snapshot could be very large, and might therefore require a long time to consolidate. Did the consolidation complete successfully, or did it end with an error message?

If the base virtual disk was thin provisioned, deleting the snapshot will likely require additional disk space on the datastore. What's the size of the VM's files (please run ls -lisa from the command line), and how much free disk space do you have on the datastore?

André

[Gysh]

Hi thanks for Answering.

1) The machine is running as of now, there is no snapshot left on the server but yet still runs on the -0000001.vmdk files

2) There is 2.47TB free on the datastore while the VMfiles uses about 1TB (if we include both normal VMDK and -0000001.vmdk files

3) The consolidation went successful without any errors and the snapshot deletion went' #1 too

As for the ls -lisa i'm not sure where to run it, since the VMware files are store on a NAS that the datastore is linked on the Vmware host

[a_p_]

The machine is running as of now, there is no snapshot left on the server but yet still runs on the -0000001.vmdk files

The "<vmname>-000001.vmdk" is the snapshot. If something goes wrong with deleting a snapshot, the snapshot may not show up in the Snapshot Manager anymore, but it still exists.

Just a quick question. Did you already try to create a new snapshot, and then click "Delete All" from the Snapshot Manger? This will delete/consolidate all snapshots, no matter whether they show up in the Snapshot Manager.

If this doesn't help getting rid of the snapshot, run the ls -lisa command in the VM's folder on the ESXi host and post the output.

André

[Gysh]

Yes I tried the delete all and it deleted the snapshots but still using the -0000001.vmdk

ls -lisa output

total 399239696

76218438      4 drwxr-xr-x    2 nfsnobod nfsnobod      4096 Dec  5 16:43 .

76218371      4 drwxrwxrwx   11 root     root          4096 Dec  5 04:37 ..

76218448      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-4f008b0400000000

76218465      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-53008b0400000000

76218404      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-54008b0400000000

76218485      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-55008b0400000000

76218464      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-5e008b0400000000

76218492      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-63008b0400000000

76218503      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-68008b0400000000

76218490      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-6b008b0400000000

76218491      4 -rwxrwxr-x    1 nfsnobod nfsnobod        84 Jan 11 22:30 .lck-76008b0400000000

76218451 49710192 -rw-------    1 nfsnobod nfsnobod 98784247808 Jan 11 22:30 machine-name-000001-flat.vmdk

76218552      4 -rw-------    1 nfsnobod nfsnobod       556 Dec  5 14:17 machine-name-000001.vmdk

76218472      0 -rw-------    1 nfsnobod nfsnobod 34359738368 Dec  5 13:40 machine-name-48986378.vswp

76218442      4 -rw-r--r--    1 nfsnobod nfsnobod        13 Dec  5 14:17 machine-name-aux.xml

76218488 32309216 -rw-------    1 nfsnobod nfsnobod 98784247808 Dec  5 13:39 machine-name-flat.vmdk

76218481     12 -rw-------    1 nfsnobod nfsnobod      8684 Jan  1 05:14 machine-name.nvram

76218544      4 -rw-------    1 nfsnobod nfsnobod       549 Dec  5 13:36 machine-name.vmdk

76218507      4 -rw-------    1 nfsnobod nfsnobod        43 Dec  5 14:17 machine-name.vmsd

76218487      8 -rw-------    1 nfsnobod nfsnobod      4314 Dec  5 14:17 machine-name.vmx

76218462      0 -rw-------    1 nfsnobod nfsnobod         0 Dec  5 13:40 machine-name.vmx.lck

76218473      4 -rw-------    1 nfsnobod nfsnobod      4082 Dec  5 13:41 machine-name.vmxf

76218486 14414148 -rw-------    1 nfsnobod nfsnobod 274877906944 Dec  5 13:41 machine-name_3-000001-flat.vmdk

76218538      4 -rw-------    1 nfsnobod nfsnobod       558 Dec  5 14:17 machine-name_3-000001.vmdk

76218461 14414148 -rw-------    1 nfsnobod nfsnobod 274877906944 Dec  5 13:39 machine-name_3-flat.vmdk

76218494      4 -rw-------    1 nfsnobod nfsnobod       551 Dec  5 13:36 machine-name_3.vmdk

76218452  46660 -rw-------    1 nfsnobod nfsnobod 34359738368 Dec  5 13:41 machine-name_6-000001-flat.vmdk

76218512      4 -rw-------    1 nfsnobod nfsnobod       556 Dec  5 14:17 machine-name_6-000001.vmdk

76218477  46660 -rw-------    1 nfsnobod nfsnobod 34359738368 Dec  5 13:39 machine-name_6-flat.vmdk

76218522      4 -rw-------    1 nfsnobod nfsnobod       549 Dec  5 13:36 machine-name_6.vmdk

76218475 186415040 -rw-------    1 nfsnobod nfsnobod 549755813888 Jan 11 22:29 machine-name_7-000001-flat.vmdk

76218511      4 -rw-------    1 nfsnobod nfsnobod       560 Dec  5 14:23 machine-name_7-000001.vmdk

76218450 86037672 -rw-------    1 nfsnobod nfsnobod 549755813888 Dec  5 13:39 machine-name_7-flat.vmdk

76218482      4 -rw-------    1 nfsnobod nfsnobod       501 Dec  5 13:36 machine-name_7.vmdk

76218467 217540 -rw-------    1 nfsnobod nfsnobod 137438953472 Jan 11 21:56 machine-name_8-000001-flat.vmdk

76218513      4 -rw-------    1 nfsnobod nfsnobod       559 Dec  5 16:43 machine-name_8-000001.vmdk

76218449 130500 -rw-------    1 nfsnobod nfsnobod 137438953472 Dec  5 13:39 machine-name_8-flat.vmdk

76218470      4 -rw-------    1 nfsnobod nfsnobod       500 Dec  5 13:36 machine-name_8.vmdk

76218453 14718780 -rw-------    1 nfsnobod nfsnobod 274877906944 Jan 11 22:30 machine-name_9-000001-flat.vmdk

76218509      4 -rw-------    1 nfsnobod nfsnobod       559 Dec  5 14:17 machine-name_9-000001.vmdk

76218454 775964 -rw-------    1 nfsnobod nfsnobod 274877906944 Dec  5 13:39 machine-name_9-flat.vmdk

76218476      4 -rw-------    1 nfsnobod nfsnobod       500 Dec  5 13:36 machine-name_9.vmdk

76218480    248 -rw-------    1 nfsnobod nfsnobod    252470 Aug  8 10:43 vmware-14.log

76218466    488 -rw-------    1 nfsnobod nfsnobod    493575 Aug 12 16:10 vmware-15.log

76218458    284 -rw-------    1 nfsnobod nfsnobod    283154 Aug 13 02:26 vmware-16.log

76218455   1060 -rw-------    1 nfsnobod nfsnobod   1080897 Dec  5 04:32 vmware-17.log

76218474    268 -rw-------    1 nfsnobod nfsnobod    272002 Dec  5 04:35 vmware-18.log

76218468    268 -rw-------    1 nfsnobod nfsnobod    272452 Dec  5 13:39 vmware-19.log

76218460    436 -rw-------    1 nfsnobod nfsnobod    441625 Jan 11 21:11 vmware.log

76218447      0 -rw-------    1 nfsnobod nfsnobod 179306496 Dec  5 13:40 vmx-machine-name-1217946488-1.vswp

Please note that after pasting the output I changed the machine name for reasons 

[Gysh]

Also please note that the machine # of HDD is valid, this server has like 5-6 HDD

[a_p_]

It looks like something went wrong on Dec. 5th. Only some of the .vmdk files have a current time stamp (ie. some of them don't seem to be in use!?), and the all of them end with -000001-flat.vmdk, although a snapshot is usually named -000001-delta. vmdk. Please take a look at the vmware*.log files to see whether they contain any hints on what went wring.

In case the -000001.vmdk header/descriptor .vmdk files do not contain a "parentFileNameHint" entry, but contain the virtual disks's geometry, then these files seem to actually be base disks (with a wrong name though).

André

[Gysh]

Hi, a.p the december 5 date is when I tried to do the snapshot create/consolidation/delete all.  I was just really busy and sadly couldn't work on the project. The problem was there before december 5.

Also After checking the logs I don't see anything strange BUT  in none of my logs i cannot search for parentFileNameHint  (says no results)

Does that means that my disks are the base disks and i'm safe?

[a_p_]

The "parentFileNameHint" entry is not in the log files, but in the .vmdk descriptor files in case these are snapshot files). Did you find the "...-000001-flat.vmdk" file names in the old logs, or when was the first time they shows up?

It would be helpful if you could attach the VM's latest vmware.log file to a reply post.

André

[Kahonu84]

Have you considered running Converter hoping the new VM will be clean??

[admin]

Yes Standalone converter can be used clone this VM to get vm without snapshot disks.

[Gysh]

Here is the most recent log

http://pastebin.com/0yry3UH0‌

I changed server name and IP adress for security purpose

Also some points to note

1) In the Datastore folder there is no -flat disk

Also if i use converter, can I

  a) Convert it on the same disk?

b) Keep the mac address while converting?

[admin]

Yes  using converter you have have thin or think disk type.

You can have static MAC to the virtual machine as well.

[a_p_]

Using Storage vMotion or the Converter - as mentioned by the other guys - is probably the safest way to rename the virtual disk files. According to the log file it indeed locks like the base disk have "incorrect" file names, which lets them look like snapshots.

1) In the Datastore folder there is no -flat disk

You cannot see these files in the Datastore Browser, but according to your file list and the log file they exist.

André