Enthusiast
Enthusiast

Yellow snapshot consolidation warning received on main production server

Jump to solution

Hi,

I have received a yellow triggered alarm as a result of trying to delete a snapshot.

Capture.PNG

This has occurred on our main production server so I am very wary about deleting anything that may cause data loss.
Within the directory I see the following files
[root@perf-esx3:/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb] ls
perfweb-000001-sesparse.vmdk

perfweb.vmsd
perfweb-000001.vmdk

perfweb.vmx
perfweb-aux.xml perfweb_5-flat.vmdk
perfweb-flat.vmdk perfweb_5.vmdk

[root@perf-esx3:/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb] ls -l *vmdk
-rw------- 1 root root 1102057472 Nov 10 11:01 perfweb-000001-sesparse.vmdk
-rw------- 1 root root 311 Nov 10 10:46 perfweb-000001.vmdk
-rw------- 1 root root 274877906944 Nov 10 10:39 perfweb-flat.vmdk
-rw------- 1 root root 498 Nov 10 10:35 perfweb.vmdk
-rw------- 1 root root 146262720512 Nov 10 11:02 perfweb_5-flat.vmdk
-rw------- 1 root root 633 Nov 10 10:46 perfweb_5.vmdk

The gui sees no snapshots that need deleting which I guess is what has prompted the alarm ( as there are files there that indicate a snapshot still exists)

Capture.PNG

Any ideas for the best course of action?

Thanks

Paul

 

Labels (1)
0 Kudos
1 Solution

Accepted Solutions
Hot Shot
Hot Shot

@paul_xedos The .vmx~ and vmx.lck files are normal when a VM is running. You shouldn't have to worry about that.

 

As far as I can judge from the information at hand, if the VM has one disk, which it does as far as I can see, we can see a couple of things:

  • A delta file is available.
  • The VM is powered ON, which explains the present lock files.

Did you already use a "Consolidate" action on the VM? Or only a "Delete All Snapshots" action? I suggest to try the Consolidate step one more time.

Visit my blog at https://vcloudvision.com!

View solution in original post

0 Kudos
5 Replies
Enthusiast
Enthusiast

Looking at vmware.log reveals that

 

2020-11-10T10:46:34.891Z| vcpu-0| A100: ConfigDB: Setting scsi0:1.redo = ""
2020-11-10T10:46:34.891Z| vcpu-0| I125: DISK: OPEN scsi0:1 '/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-000001.vmdk' persistent R[]
2020-11-10T10:46:34.892Z| vcpu-0| I125: DISKLIB-VMFS : "/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-000001-sesparse.vmdk" : open successful (10) size = 1102057472, hd = 39438210. Type 19
2020-11-10T10:46:34.892Z| vcpu-0| I125: DISKLIB-DSCPTR: Opened [0]: "perfweb-000001-sesparse.vmdk" (0xa)
2020-11-10T10:46:34.892Z| vcpu-0| I125: DISKLIB-LINK : Opened '/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-000001.vmdk' (0xa): seSparse, 536870912 sectors / 256 GB.
2020-11-10T10:46:34.893Z| vcpu-0| I125: DISKLIB-VMFS : "/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-flat.vmdk" : open successful (14) size = 274877906944, hd = 44943235. Type 3
2020-11-10T10:46:34.893Z| vcpu-0| I125: DISKLIB-DSCPTR: Opened [0]: "perfweb-flat.vmdk" (0xe)
2020-11-10T10:46:34.893Z| vcpu-0| I125: DISKLIB-LINK : Opened '/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb.vmdk' (0xe): vmfs, 536870912 sectors / 256 GB.
2020-11-10T10:46:34.893Z| vcpu-0| I125: DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 44943235, extentType = 2
2020-11-10T10:46:34.893Z| vcpu-0| I125: DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 39438210, extentType = 1
2020-11-10T10:46:34.899Z| vcpu-0| I125: DISKLIB-LIB : Opened "/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-000001.vmdk" (flags 0xa, type vmfs).
2020-11-10T10:46:34.899Z| vcpu-0| I125: DISK: Disk '/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-000001.vmdk' has UUID '60 00 c2 99 98 c6 0d ed-ae de 52 30 eb fd 14 5d'
2020-11-10T10:46:34.899Z| vcpu-0| I125: DISK: OPEN '/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-000001.vmdk' Geo (33418/255/63) BIOS Geo (0/0/0)
2020-11-10T10:46:34.899Z| vcpu-0| I125: Creating virtual dev for 'scsi0:1'.
2020-11-10T10:46:34.899Z| vcpu-0| I125: DumpDiskInfo: scsi0:1 createType=11, capacity = 536870912, numLinks = 2, allocationType = 2
2020-11-10T10:46:34.899Z| vcpu-0| I125: SCSIDiskESXPopulateVDevDesc: Using FS backend
2020-11-10T10:46:34.899Z| vcpu-0| I125: DISKUTIL: scsi0:1 : geometry=33418/255/63
2020-11-10T10:46:34.899Z| vcpu-0| I125: SCSIFilterESXAttachCBRCInt: CBRC not enabled or opened without filters,skipping CBRC filter attach.
2020-11-10T10:46:34.899Z| vcpu-0| I125: SCSIFilterSBDAttachCBRC: device scsi0:1 is not SBD. Skipping CBRC attach SBD way.
2020-11-10T10:46:34.899Z| vcpu-0| I125: VigorTransport_ServerSendResponse opID=bb05f2ae-6d90-4cbb-b3bf-348ee09590a3-27843-auto-27844-h5c:70004008-50-61-410e seq=1768590: Completed Snapshot request with messages.
2020-11-10T10:46:34.899Z| vcpu-0| I125: Turning off snapshot info cache.
2020-11-10T10:46:34.900Z| vcpu-0| I125: Turning off snapshot disk cache.
2020-11-10T10:46:34.900Z| vcpu-0| I125: ConsolidateEnd: Snapshot consolidate complete: Failed to lock the file (5).
2020-11-10T10:46:34.900Z| vcpu-0| I125: Vix: [mainDispatch.c:4239]: VMAutomation_ReportPowerOpFinished: statevar=3, newAppState=1881, success=1 additionalError=0

2020-11-10T10:46:36.165Z| vcpu-3| I125: HBACommon: First write on scsi0:1.fileName='/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb-000001.vmdk'
2020-11-10T10:46:36.165Z| vcpu-3| I125: DDB: "longContentID" = "1917db2632e3f2dd3f566138c7b6d078" (was "939f5becd1ce7311bdab2fdf4ab76edc")
2020-11-10T10:46:36.186Z| vcpu-3| I125: DISKLIB-CHAIN : DiskChainUpdateContentID: old=0x4ab76edc, new=0xc7b6d078 (1917db2632e3f2dd3f566138c7b6d078)
2020-11-10T10:46:42.228Z| vcpu-0| I125: HBACommon: First write on scsi0:0.fileName='/vmfs/volumes/5d5d328a-1c1803fd-b216-00c0dd175878/perfweb/perfweb_5.vmdk'
2020-11-10T10:46:42.229Z| vcpu-0| I125: DDB: "longContentID" = "089c7d212e31c24f71c0ab74494b10f8" (was "68d5d06e36abf43022a6a23a87dfee95")
2020-11-10T10:46:42.245Z| vcpu-0| I125: DISKLIB-CHAIN : DiskChainUpdateContentID: old=0x87dfee95, new=0x494b10f8 (089c7d212e31c24f71c0ab74494b10f8)

 

I'm not sure which lck file it sees but the only one now showing is this

-rwx------ 1 root root 3782 Nov 10 10:46 perfweb.vmx
-rw------- 1 root root 0 Oct 8 11:25 perfweb.vmx.lck
-rwx------ 1 root root 3789 Nov 10 10:46 perfweb.vmx~

Worryingly, the vmx file with the ~ symbol looks like its been opened by the snapshotting process and changed

Some further info which I am sure is relevant is that half an hour before I attempted to delete a snapshot via the gui. This sometimes takes quite a few minutes so I went away. When I came back there was no progress bar showing snapshot deletion status but the snapshot still existed. I therefore re-submitted the snapshot delete command a second time. Presumably this is why I now see the lock issue

0 Kudos
Hot Shot
Hot Shot

@paul_xedos The .vmx~ and vmx.lck files are normal when a VM is running. You shouldn't have to worry about that.

 

As far as I can judge from the information at hand, if the VM has one disk, which it does as far as I can see, we can see a couple of things:

  • A delta file is available.
  • The VM is powered ON, which explains the present lock files.

Did you already use a "Consolidate" action on the VM? Or only a "Delete All Snapshots" action? I suggest to try the Consolidate step one more time.

Visit my blog at https://vcloudvision.com!

View solution in original post

0 Kudos
Enthusiast
Enthusiast

Perfect that was the fix. Not sure what consolidate does that deleting a snapshot does not but I'm very glad it did. Thanks for your help

0 Kudos
Hot Shot
Hot Shot

@paul_xedos No problem! Like I said the Consolidate just removes the redundant redo/delta files. A delete snapshot should normally do that, but sometimes it fails and that is why the Consolidate action was created.

Visit my blog at https://vcloudvision.com!
Virtuoso
Virtuoso

Well....  the consolidation warning/alarm is triggert as soon as a not so consistent state is detected. From your information the VMs used multiple vDisks (i can see a "_5") and only some of the disk are running on a snaphot. But the Snapshot Database doesnt contains a entry anymore.

Without an entry within the Database (simple ascii textfile) you cant use the delete function and this is when the consolidate comes into play. It tries to delete every snapshot regards of whats  vCenter think.

I see this very often on VMs with multiple (large) vDisk. You can check this very fast if you try to edit a VM and check the vDisks sizes. vDisks with snaps are greyed out and only vDisks without a snap are resizable.

Regards,
Joerg