Can anyone tell me what this tells you?
I have never seen a snapshot state like this.
I can click on the "u" state and select "go to", delete or delete-all
What happens is if I click either of these?
I am having the idea that I am on the latest state, but the logically the image tell me something else.
Regards, Lars.
That is not really a weird state.
It is also not necessarily an indication of serious corruption.
Only important step to take now is change your standard tool for the next necessary steps.
Very likely you will know that the vmsd-file is responsible for the creation of the snapshot-map in Snapshot-manager.
VMSD-files are fragile. Good thing: Snapshot-manager display is used only for UI-actions. It is not necessary for manual snapshot-actions.
Long story short: assume your VMSD is at least partially damaged - so ignore it.
Very likely you have 2 or more branches of snapshots.
u - represents a chain like: basedisk-snap*-snap*-snap*... there may just be one base-000001.vmdk - but there could be hundreds
you are here - represents a different chain: basedisk-snapX*.....
Your task is to find out if any of the existing branches has the newest production data.
If the delta.vmdk with the newest timestamp has the latest data -create a full cone of that state via vmkfstools -i messedup-snapshots.vmdk consolidated.vmdk
Watch for gaps: it is possible to get a VM with latest data, a working guestOS but 3 years of missing data.
Inspect all vmdk-descriptorfiles you have.
Read the logs that still have the last successful operations captured.
Ask colleagues if anybody edited the vmx-file.
To assist you we need a detailed fillist, all descriptorfiles and all vmware.logs and the vmsd-file.
Thank you for helping... and sorry that I was so late getting back. We actually have two guests with the same issue.
The first one (containing an independent disk) looks like this:
-rw------- 1 root root 14495739904 May 26 21:16 sbssrv-000001-delta.vmdk
-rw------- 1 root root 316 May 26 09:13 sbssrv-000001.vmdk
-rw------- 1 root root 33779712 May 12 20:18 sbssrv-000002-delta.vmdk
-rw------- 1 root root 316 May 12 20:18 sbssrv-000002.vmdk
-rw-r--r-- 1 root root 27 Oct 15 2012 sbssrv-44958a36.hlog
-rw------- 1 root root 4246732800 May 26 09:13 sbssrv-44958a36.vswp
-rw------- 1 root root 33073 May 12 20:17 sbssrv-Snapshot331.vmsn
-rw-r--r-- 1 root root 127 May 12 20:18 sbssrv-aux.xml
-rw------- 1 root root 117037858816 May 12 20:17 sbssrv-flat.vmdk
-rwx------ 1 root root 255820 May 12 20:18 sbssrv-vss_manifests331.zip
-rw------- 1 root root 8684 May 26 09:14 sbssrv.nvram
-rw------- 1 root root 495 May 12 20:16 sbssrv.vmdk
-rw-r--r-- 1 root root 716 May 22 22:02 sbssrv.vmsd
-rwxr-xr-x 1 root root 4921 May 26 20:27 sbssrv.vmx
-rw------- 1 root root 0 May 26 09:13 sbssrv.vmx.lck
-rw-r--r-- 1 root root 2932 May 22 20:02 sbssrv.vmxf
-rwxr-xr-x 1 root root 4922 May 26 20:27 sbssrv.vmx~
-rw------- 1 root root 16822272 May 26 09:14 sbssrv_1-000001-delta.vmdk
-rw------- 1 root root 319 May 26 09:14 sbssrv_1-000001.vmdk
-rw------- 1 root root 16822272 May 12 20:18 sbssrv_1-000002-delta.vmdk
-rw------- 1 root root 319 May 12 20:18 sbssrv_1-000002.vmdk
-rw------- 1 root root 21474836480 May 12 20:17 sbssrv_1-flat.vmdk
-rw------- 1 root root 495 May 12 20:17 sbssrv_1.vmdk
-rw------- 1 root root 1980022784 May 26 20:55 sbssrv_2-000001-delta.vmdk
-rw------- 1 root root 320 May 26 09:14 sbssrv_2-000001.vmdk
-rw------- 1 root root 17088512 May 12 20:18 sbssrv_2-000002-delta.vmdk
-rw------- 1 root root 320 May 12 20:18 sbssrv_2-000002.vmdk
-rw------- 1 root root 161061273600 May 12 20:17 sbssrv_2-flat.vmdk
-rw------- 1 root root 497 May 12 20:16 sbssrv_2.vmdk
-rw-r--r-- 1 root root 50775 Nov 7 2013 vmware-31.log
-rw-r--r-- 1 root root 50775 Nov 7 2013 vmware-32.log
-rw-r--r-- 1 root root 50718 Nov 7 2013 vmware-33.log
-rw-r--r-- 1 root root 334125 Nov 15 2013 vmware-34.log
-rw-r--r-- 1 root root 1497650357 Nov 25 2015 vmware-35.log
-rw-r--r-- 1 root root 130964136 May 26 09:12 vmware-36.log
-rw-r--r-- 1 root root 206024 May 26 20:28 vmware.log
-rw------- 1 root root 112197632 May 26 09:13 vmx-sbssrv-1150650934-1.vswp
cat sbssrv:
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "7"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
nvram = "sbssrv.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "soft"
displayName = "sbssrv"
extendedConfigFile = "sbssrv.vmxf"
floppy0.present = "TRUE"
scsi0.present = "TRUE"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsisas1068"
scsi1.present = "TRUE"
scsi1.sharedBus = "none"
scsi1.virtualDev = "lsisas1068"
memsize = "6144"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "sbssrv-000001.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
scsi1:0.present = "TRUE"
scsi1:0.fileName = "sbssrv_2-000001.vmdk"
scsi1:0.deviceType = "scsi-hardDisk"
scsi1:1.present = "TRUE"
scsi1:1.fileName = "sbssrv_1-000001.vmdk"
scsi1:1.deviceType = "scsi-hardDisk"
ide1:0.present = "TRUE"
ide1:0.clientDevice = "TRUE"
ide1:0.deviceType = "cdrom-raw"
ide1:0.startConnected = "FALSE"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.clientDevice = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "FT Network"
ethernet0.addressType = "generated"
chipset.onlineStandby = "FALSE"
disk.EnableUUID = "TRUE"
guestOS = "windows7srv-64"
uuid.location = "56 4d 97 e4 55 a0 25 58-8e e3 ed cd 5b 91 ca 80"
uuid.bios = "56 4d 97 e4 55 a0 25 58-8e e3 ed cd 5b 91 ca 80"
vc.uuid = "52 f3 f4 99 96 f4 c1 91-b9 3c b4 7b a0 a9 60 df"
snapshot.action = "keep"
sched.cpu.min = "0"
sched.cpu.units = "mhz"
sched.cpu.shares = "normal"
sched.mem.minsize = "2094"
sched.mem.shares = "normal"
ethernet0.generatedAddress = "00:0c:29:91:ca:80"
svga.vramSize = "8388608"
vmci0.id = "1536281216"
cleanShutdown = "FALSE"
replay.supported = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/4d62be23-e060c5f4-bc2a-001e6707858d/sbssrv/sbssrv-44958a36.vswp"
replay.filename = ""
scsi0:0.redo = ""
scsi1:0.redo = ""
scsi1:1.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
scsi1.pciSlotNumber = "192"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "33"
scsi0.sasWWID = "50 05 05 64 55 a0 25 50"
scsi1.sasWWID = "50 05 05 64 55 a0 24 50"
vmotion.checkpointFBSize = "8388608"
ethernet0.generatedAddressOffset = "0"
hostCPUID.0 = "0000000b756e65476c65746e49656e69"
hostCPUID.1 = "000206c200200800029ee3ffbfebfbff"
hostCPUID.80000001 = "0000000000000000000000012c100800"
guestCPUID.0 = "0000000b756e65476c65746e49656e69"
guestCPUID.1 = "000206c200010800829822030fabfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000b756e65476c65746e49656e69"
userCPUID.1 = "000206c200200800029822030fabfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"
tools.remindInstall = "TRUE"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
tools.syncTime = "FALSE"
unity.wasCapable = "TRUE"
ide1:0.fileName = "/usr/lib/vmware/isoimages/windows.iso"
sched.cpu.affinity = "all"
usb.present = "TRUE"
migrate.hostlog = "./sbssrv-44958a36.hlog"
ehci.present = "TRUE"
usb.pciSlotNumber = "34"
ehci.pciSlotNumber = "35"
usb:1.present = "TRUE"
usb:1.deviceType = "hub"
usb:1.speed = "2"
softPowerOff = "FALSE"
config.readOnly = "FALSE"
disk.AllowDupUUID = "TRUE"
usb.autoConnect.device0 = "path:7/0 autoclean:1"
ethernet1.present = "FALSE"
scsi0:1.present = "TRUE"
scsi0:2.present = "FALSE"
scsi0:3.present = "FALSE"
scsi0:4.present = "FALSE"
scsi0:1.fileName = "/vmfs/volumes/8fd0b324-a7eb2c0e/sbssrv/sbssrv_3.vmdk"
scsi0:1.mode = "independent-persistent"
scsi0:1.ctkEnabled = "FALSE"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:1.redo = ""
sched.scsi0:1.shares = "normal"
sched.scsi0:1.throughputCap = "off"
scsi0:2.fileName = "/vmfs/volumes/d1dcb19a-b3c9a4d9/Firewal_replica_1/Firewal-000004.vmdk"
scsi0:2.mode = "persistent"
scsi0:2.ctkEnabled = "FALSE"
scsi0:2.deviceType = "scsi-hardDisk"
scsi0:2.redo = ""
scsi0:3.fileName = "/vmfs/volumes/4d62be23-e060c5f4-bc2a-001e6707858d/Firewal/Firewal.vmdk"
scsi0:3.mode = "independent-nonpersistent"
scsi0:3.ctkEnabled = "FALSE"
scsi0:3.deviceType = "scsi-hardDisk"
scsi0:3.redo = ""
scsi0:4.fileName = "/vmfs/volumes/d1dcb19a-b3c9a4d9/sbssrv_replica_1/sbssrv_1-000006.vmdk"
scsi0:4.mode = "persistent"
scsi0:4.ctkEnabled = "FALSE"
scsi0:4.deviceType = "scsi-hardDisk"
scsi0:4.redo = ""
usb:0.present = "TRUE"
usb:0.deviceType = "mouse"
usb:0.port = "0"
usb:0.parent = "-1"
The next one looks like this:
-rw------- 1 root root 3146240 May 26 09:10 sqlsrv-000001-ctk.vmdk
-rw------- 1 root root 3640758272 May 26 21:25 sqlsrv-000001-delta.vmdk
-rw------- 1 root root 381 May 26 09:09 sqlsrv-000001.vmdk
-rw------- 1 root root 3146240 May 12 19:08 sqlsrv-000002-ctk.vmdk
-rw------- 1 root root 33656832 May 12 19:08 sqlsrv-000002-delta.vmdk
-rw------- 1 root root 381 May 12 19:08 sqlsrv-000002.vmdk
-rw------- 1 root root 10588520448 May 26 09:09 sqlsrv-3ddf619b.vswp
-rw------- 1 root root 32761 May 12 19:06 sqlsrv-Snapshot51.vmsn
-rw-r--r-- 1 root root 125 May 12 19:08 sqlsrv-aux.xml
-rw------- 1 root root 3146240 May 12 19:06 sqlsrv-ctk.vmdk
-rw------- 1 root root 51539607552 May 12 19:06 sqlsrv-flat.vmdk
-rwx------ 1 root root 212373 May 12 19:09 sqlsrv-vss_manifests51.zip
-rw------- 1 root root 8684 May 26 09:10 sqlsrv.nvram
-rw------- 1 root root 575 May 7 21:04 sqlsrv.vmdk
-rw------- 1 root root 570 May 21 21:56 sqlsrv.vmsd
-rw------- 1 root root 3807 May 26 09:09 sqlsrv.vmx
-rw------- 1 root root 0 May 26 09:09 sqlsrv.vmx.lck
-rw------- 1 root root 3827 Nov 25 2015 sqlsrv.vmxf
-rw------- 1 root root 3807 May 26 09:09 sqlsrv.vmx~
-rw------- 1 root root 983552 May 26 09:10 sqlsrv_1-000001-ctk.vmdk
-rw------- 1 root root 1442873344 May 26 21:17 sqlsrv_1-000001-delta.vmdk
-rw------- 1 root root 386 May 26 09:10 sqlsrv_1-000001.vmdk
-rw------- 1 root root 983552 May 12 19:08 sqlsrv_1-000002-ctk.vmdk
-rw------- 1 root root 16809984 May 12 19:08 sqlsrv_1-000002-delta.vmdk
-rw------- 1 root root 386 May 12 19:08 sqlsrv_1-000002.vmdk
-rw------- 1 root root 983552 May 12 19:06 sqlsrv_1-ctk.vmdk
-rw------- 1 root root 16106127360 May 12 19:06 sqlsrv_1-flat.vmdk
-rw------- 1 root root 578 May 7 21:14 sqlsrv_1.vmdk
-rw------- 1 root root 983552 May 26 09:10 sqlsrv_2-000001-ctk.vmdk
-rw------- 1 root root 1107329024 May 26 21:23 sqlsrv_2-000001-delta.vmdk
-rw------- 1 root root 386 May 26 09:10 sqlsrv_2-000001.vmdk
-rw------- 1 root root 983552 May 12 19:08 sqlsrv_2-000002-ctk.vmdk
-rw------- 1 root root 16809984 May 12 19:08 sqlsrv_2-000002-delta.vmdk
-rw------- 1 root root 386 May 12 19:08 sqlsrv_2-000002.vmdk
-rw------- 1 root root 983552 May 12 19:06 sqlsrv_2-ctk.vmdk
-rw------- 1 root root 16106127360 May 12 19:06 sqlsrv_2-flat.vmdk
-rw------- 1 root root 578 May 7 21:04 sqlsrv_2.vmdk
-rw------- 1 root root 582794 Nov 9 2014 vmware-1.log
-rw------- 1 root root 3934811 Nov 25 2015 vmware-2.log
-rw------- 1 root root 4756139 May 26 09:09 vmware-3.log
-rw------- 1 root root 212372 May 26 20:58 vmware.log
-rw------- 1 root root 132120576 May 26 09:09 vmx-sqlsrv-1038049691-1.vswp
cat sqlsrv.vmx
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "7"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
nvram = "sqlsrv.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "soft"
displayName = "sqlsrv"
extendedConfigFile = "sqlsrv.vmxf"
floppy0.present = "TRUE"
scsi0.present = "TRUE"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsisas1068"
scsi1.present = "TRUE"
scsi1.sharedBus = "none"
scsi1.virtualDev = "lsisas1068"
memsize = "12288"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "sqlsrv-000001.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
scsi1:0.present = "TRUE"
scsi1:0.fileName = "sqlsrv_1-000001.vmdk"
scsi1:0.deviceType = "scsi-hardDisk"
scsi1:1.present = "TRUE"
scsi1:1.fileName = "sqlsrv_2-000001.vmdk"
scsi1:1.deviceType = "scsi-hardDisk"
ide1:0.present = "TRUE"
ide1:0.clientDevice = "TRUE"
ide1:0.deviceType = "cdrom-raw"
ide1:0.startConnected = "FALSE"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.clientDevice = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
ethernet0.networkName = "FT Network"
ethernet0.addressType = "generated"
chipset.onlineStandby = "FALSE"
disk.EnableUUID = "TRUE"
guestOS = "windows7srv-64"
uuid.location = "56 4d c6 d8 7f 17 d5 3c-74 98 d2 3a 88 08 8b 86"
uuid.bios = "56 4d a8 bb 50 28 31 c8-b5 bb 86 56 28 e2 55 33"
vc.uuid = "52 51 6f aa ed c9 4b cc-6e 60 5e eb 2d 1b 5a 1b"
snapshot.action = "keep"
sched.cpu.min = "0"
sched.cpu.units = "mhz"
sched.cpu.shares = "normal"
sched.mem.minsize = "2190"
sched.mem.shares = "normal"
sched.cpu.affinity = "all"
ethernet0.generatedAddress = "00:0c:29:e2:55:33"
svga.vramSize = "8388608"
vmci0.id = "685921587"
tools.syncTime = "FALSE"
cleanShutdown = "FALSE"
replay.supported = "FALSE"
unity.wasCapable = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/52f827de-473c0ec6-0216-a0d3c1f3802c/sqlsrv/sqlsrv-3ddf619b.vswp"
replay.filename = ""
scsi0:0.redo = ""
scsi1:0.redo = ""
scsi1:1.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
scsi0.pciSlotNumber = "160"
scsi1.pciSlotNumber = "192"
ethernet0.pciSlotNumber = "32"
vmci0.pciSlotNumber = "33"
scsi0.sasWWID = "50 05 05 6b 50 28 31 c0"
scsi1.sasWWID = "50 05 05 6b 50 28 30 c0"
vmotion.checkpointFBSize = "8388608"
ethernet0.generatedAddressOffset = "0"
hostCPUID.0 = "0000000d756e65476c65746e49656e69"
hostCPUID.1 = "000306e40020080077bee3ffbfebfbff"
hostCPUID.80000001 = "0000000000000000000000012c100800"
guestCPUID.0 = "0000000d756e65476c65746e49656e69"
guestCPUID.1 = "000306e400010800829822030fabfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000d756e65476c65746e49656e69"
userCPUID.1 = "000306e400010800829822030fabfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"
ide1:0.fileName = "/usr/lib/vmware/isoimages/windows.iso"
softPowerOff = "FALSE"
toolsInstallManager.lastInstallError = "0"
toolsInstallManager.updateCounter = "2"
tools.remindInstall = "FALSE"
ctkEnabled = "TRUE"
scsi0:0.ctkEnabled = "TRUE"
scsi1:1.ctkEnabled = "TRUE"
scsi1:0.ctkEnabled = "TRUE"
sched.scsi0:0.throughputCap = "off"
sched.scsi0:0.shares = "normal"
scsi0:1.deviceType = "scsi-hardDisk"
scsi0:2.deviceType = "scsi-hardDisk"
scsi0:3.deviceType = "scsi-hardDisk"
I am pretty sure no one has been editing the vmx's by hand.
Thank you for helping.
Regards, Lars.
My guess is BTW that the snapshots got broken somehow by our VEEAM solution (which BTW are now failing on these two servers.
No one to comment on this?
Planning on doing a vmware convert into a new guest... maybe it is not needed (?), but I am afraid to mess around with the snapshots in case something gets even more screwed up.
Regards, Lars.
What exactly puzzles you ?
I cant find anything alarming in your details ?
Really - that's good.
What puzzles me is that I cannot take further snapshots... and I am not sure how to delete the snapshots already there. When I mark the "u" snapshot, I can select "Delete" (even Delete All and Go to),but I find the state of snapshot more weird than I have seen anywhere else.
I have two delta files for the drives - but only one snapshot I can go to?
I want to continue from the state the virtual machine is actually running in and delete all snapshots taken.
Regards, Lars.