Re: Virtual Machine reverted back in time after DS...

jurroen · ‎01-03-2020

Hi,

I am experiencing an issue for the second time now, so I wanted to ask help. I use ESXi for a couple of VMs, version 6.7.0 Update 3 (Build 15018017). After I move a VM to a different datastore (from a VMFS5 datastore to a VMFS6 datastore), it somehow uses a disk of months ago - like the past X months weren't stored properly.

What I did to migrate:

Shutdown the VM
Make sure there are no snapshots - and delete snapshots if any
Unregister the VM
Use the datastore browser to move the entire VM directory to the second datastore
After completion, register the VM
Power-on the VM

After going through this process, it shows the latest files within the VM from april 2019, while there were changes every day after that. But those... just disappeared?

The VMX contents:

.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "13"
vmci0.present = "TRUE"
floppy0.present = "FALSE"
numvcpus = "2"
memSize = "16384"
bios.bootRetry.delay = "10"
tools.upgrade.policy = "manual"
sched.cpu.units = "mhz"
vm.createDate = "1556720757032586"
usb.present = "TRUE"
ehci.present = "TRUE"
scsi0.virtualDev = "lsisas1068"
scsi0.present = "TRUE"
sata0.present = "TRUE"
sata0:0.startConnected = "FALSE"
sata0:0.allowGuestConnectionControl = "FALSE"
sata0:0.deviceType = "cdrom-raw"
sata0:0.clientDevice = "TRUE"
sata0:0.fileName = "emptyBackingString"
sata0:0.present = "TRUE"
ethernet0.virtualDev = "e1000e"
ethernet0.networkName = "llc-ams01-cust-net1"
ethernet0.addressType = "static"
ethernet0.address = "00:0c:29:2c:f2:f3"
ethernet0.wakeOnPcktRcv = "FALSE"
ethernet0.present = "TRUE"
displayName = "llc-ams01-cust-vmname"
guestOS = "windows9srv-64"
toolScripts.afterPowerOn = "TRUE"
toolScripts.afterResume = "TRUE"
toolScripts.beforeSuspend = "TRUE"
toolScripts.beforePowerOff = "TRUE"
tools.syncTime = "FALSE"
uuid.bios = "56 4d b4 10 93 51 a9 5a-c0 26 e5 53 fa 05 66 c6"
uuid.location = "56 4d 97 05 db f8 07 1a-49 f5 1e 90 ea 28 99 74"
vc.uuid = "52 88 5e f4 99 50 64 cf-da b9 85 ae 6d 39 e9 26"
sched.cpu.min = "0"
sched.cpu.shares = "high"
sched.mem.min = "0"
sched.mem.minSize = "0"
sched.mem.shares = "high"
sched.cpu.affinity = "all"
vmci0.id = "-100309306"
cleanShutdown = "FALSE"
extendedConfigFile = "llc-ams01-cust-vmname.vmxf"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "llc-ams01-cust-vmname.vmdk"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
scsi0:0.present = "TRUE"
disk.EnableUUID = "TRUE"
featMask.vm.hv.capable = "Min:1"
tools.guest.desktop.autolock = "FALSE"
nvram = "llc-ams01-cust-vmname.nvram"
pciBridge0.present = "TRUE"
svga.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
hpet0.present = "TRUE"
RemoteDisplay.maxConnections = "-1"
cpuid.coresPerSocket = "2"
sched.cpu.latencySensitivity = "normal"
numa.autosize.vcpu.maxPerVirtualNode = "2"
numa.autosize.cookie = "20001"
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
usb.pciSlotNumber = "32"
ethernet0.pciSlotNumber = "192"
ehci.pciSlotNumber = "33"
vmci0.pciSlotNumber = "34"
sata0.pciSlotNumber = "35"
scsi0.sasWWID = "50 05 05 60 93 51 a9 50"
monitor.phys_bits_used = "43"
vmotion.checkpointFBSize = "4194304"
vmotion.checkpointSVGAPrimarySize = "4194304"
softPowerOff = "FALSE"
usb:1.speed = "2"
usb:1.present = "TRUE"
usb:1.deviceType = "hub"
usb:1.port = "1"
usb:1.parent = "-1"
toolsInstallManager.lastInstallError = "0"
svga.guestBackedPrimaryAware = "TRUE"
tools.remindInstall = "FALSE"
toolsInstallManager.updateCounter = "10"
ethernet0.checkMACAddress = "FALSE"
vm.genid = "-3427782277700550669"
vm.genidX = "-3089548162795944296"
nakivo.uuid = "754944da-a301-4175-b169-ace492abc3ba"
sched.swap.derivedName = "/vmfs/volumes/5d189dec-b1f98545-32a2-801844f26f5a/llc-ams01-cust-vmname/llc-ams01-cust-vmname-aa2c8000.vswp"
migrate.hostLog = "./LLC-RDP-VMNAME-AMS01-5efdd000.hlog"
scsi0:0.redo = ""
usb:0.present = "TRUE"
usb:0.deviceType = "hid"
usb:0.port = "0"
usb:0.parent = "-1"

And the contents of the VM directory:

[root@ams01vh01:~] ls -lah /vmfs/volumes/HDD/llc-ams01-cust-vmname/
total 620513600
drwxr-xr-x    1 root     root       76.0K Jan 3 12:43 .
drwxr-xr-t    1 root     root       76.0K Dec 29 12:26 ..
-rw-------    1 root     root      129.7G Dec 28 21:09 llc-ams01-cust-vmname-000001-delta.vmdk
-rw-------    1 root     root         378 Dec 28 21:15 llc-ams01-cust-vmname-000001.vmdk
-rw-------    1 root     root       44.8G Dec 28 21:15 llc-ams01-cust-vmname-000002-delta.vmdk
-rw-------    1 root     root         331 Dec 28 22:03 llc-ams01-cust-vmname-000002.vmdk
-rw-------    1 root     root       24.0G Dec 28 21:18 llc-ams01-cust-vmname-000003-delta.vmdk
-rw-------    1 root     root         331 Dec 28 22:03 llc-ams01-cust-vmname-000003.vmdk
-rw-------    1 root     root       30.6G Dec 28 21:22 llc-ams01-cust-vmname-000004-delta.vmdk
-rw-------    1 root     root         331 Dec 28 22:03 llc-ams01-cust-vmname-000004.vmdk
-rw-------    1 root     root       16.0G Jan 3 12:41 llc-ams01-cust-vmname-aa2c8000.vswp
-rw-------    1 root     root          13 Dec 28 22:03 llc-ams01-cust-vmname-aux.xml
-rw-------    1 root     root      350.0G Jan 3 13:44 llc-ams01-cust-vmname-flat.vmdk
-rw-------    1 root     root        8.5K Jan 3 12:44 llc-ams01-cust-vmname.nvram
-rw-------    1 root     root         543 Jan 3 12:43 llc-ams01-cust-vmname.vmdk
-rw-------    1 root     root          77 Dec 28 22:03 llc-ams01-cust-vmname.vmsd
-rw-------    1 root     root        3.5K Jan 3 12:41 llc-ams01-cust-vmname.vmx
-rw-------    1 root     root           0 Jan 3 12:41 llc-ams01-cust-vmname.vmx.lck
-rw-------    1 root     root        3.3K Dec 28 22:03 llc-ams01-cust-vmname.vmxf
-rw-------    1 root     root        3.4K Jan 3 12:41 llc-ams01-cust-vmname.vmx~
-rw-------    1 root     root      972.2K Dec 28 22:03 vmware-10.log
-rw-------    1 root     root      372.6K Dec 28 22:03 vmware-11.log
-rw-------    1 root     root      214.2K Jan 3 12:40 vmware-12.log
-rw-------    1 root     root        1.6M Dec 28 22:03 vmware-7.log
-rw-------    1 root     root      515.1K Dec 28 22:03 vmware-8.log
-rw-------    1 root     root      309.0K Dec 28 22:03 vmware-9.log
-rw-------    1 root     root      346.0K Jan 3 13:20 vmware.log
-rw-------    1 root     root       86.0M Dec 28 22:03 vmx-llc-ams01-cust-vmname-2227188268-2.vswp
-rw-------    1 root     root       82.0M Jan 3 12:41 vmx-llc-ams01-cust-vmname-2855043072-1.vswp

berndweyand · ‎01-03-2020

did you really delete the snapshots ? there are 4 snapshot-files visible.

check the llc-ams01-cust-vmname.vmsd to see which snapshot is valid

jurroen · ‎01-03-2020

Thank you for your incredibly swift reply, really appreciated!

I did delete the snapshots from the ESXi web interface, of that I am sure. Here is the contents of the vmsd:

[root@ams01vh01:~] cat /vmfs/volumes/HDD/llc-ams01-cust-vmname/llc-ams01-cust-vmname.vmsd
.encoding = "UTF-8"
snapshot.lastUID = "4"
snapshot.needConsolidate = "TRUE"

Can I either edit this file or delete it to not use a snapshot and fallback to the regular vmdk?

berndweyand · ‎01-03-2020

perhaps you overlooked the warning "virtual machine needs consolidation" ?

the snapshot deletion failed so you need to consolidate (rightclick->consolidate)

did you move or copy the files to the new destination ? maybe its better to have a backup before because i dont know the behaviour of the webinterface with moved and re-registered machines

a_p_ · ‎01-03-2020

The .vmsd file (that's what you also see in the UI) is not reliable, if an issue occurred with deleting snapshots. To find out whether a VM has active snapshots, always check the VM's .vmdk file names in the settings, and/or check whether any delta, or separse files exist in the datastore. Snapshot files are named like <vmname>-00000x.vmdk.

Anyway, to find out what can be done, provide some information:

for how long has the VM been powered on, when you detected the roll-back
compress/zip all of the VM's descriptor .vmdk files (the small ones), and attach the .zip archive to a reply post

André

a_p_ · ‎01-03-2020

I just re-read your initial post and found:

from a VMFS5 datastore to a VMFS6 datastore

Tha's going to be an issue, because VMFS6 doesn't support delta files anymore. You will have to move (or better copy) the VM back to a VMFS5 datastore in order to try, and fix the issue.

André

jurroen · ‎01-03-2020

perhaps you overlooked the warning "virtual machine needs consolidation" ?
the snapshot deletion failed so you need to consolidate (rightclick->consolidate)
did you move or copy the files to the new destination ? maybe its better to have a backup before because i dont know the behaviour of the webinterface with moved and re-registered machines

Thank you for you swift reply, once more.

I'll create a backup before trying to consolidate the disks. I have moved the files to their destination, rather than copying.

jurroen · ‎01-03-2020

Thank you for your reply!

for how long has the VM been powered on, when you detected the roll-back

The machine was moved on the 28th of december, has been powered off since. I just discovered the "rollback" - it is now offline again. Has been online for two hours max.

compress/zip all of the VM's descriptor .vmdk files (the small ones), and attach the .zip archive to a reply post

This might sound really newbie: is there a chance that the descriptor files contain confidential information? And if so, can I take precautions to remove that particular info?

a_p_ · ‎01-03-2020

The .vmdk descriptor files are plain text files which can be viewed using an editor (which supports Linux line breaks). They only contain metadata, like the virtual disk's geometry, the snapshot chain information (which is what I am looking for), ...

Since they are kind of hidden in VMware datastore browser, you may use e.g. WinSCP to connect to the host, and download them.

Regarding the VM's uptime. You said that the VM has been powered on since Dec, 28th, but then you also say that it's been online for only 2 hours. Can you please clarify this. The point is that chances to recover data without (or with less) corruption are better, the less modifications/writes to the base .vmdk file has occurred.

André

All

Virtual Machine reverted back in time after DS move