VMware Cloud Community
Tom31415
Enthusiast
Enthusiast

Container volume disk consolidation

Hello,

I have a problem with my container volumes, they are running from delta files, some of them have 16 delta stages and I have no idea how to consolidate these disks because I tried everything what I found on the internet. I tried, creating snapshot with delete all snapshots after that with and without quiescence checkbox. I tried migration from vsanDatastore to NFS datastore where I tried both options with snapshots but delta files are still there and nothing changed. The VM sometimes tell me warning about consolidation needed so I run it and it finished successfully but my disks are still in delta files and I still cannot change disk size. My datastore has a lot of free space. I was looking for some locks but the terminal tool tell me that there is no lock.

Delta files was probably created by my Kasten backup tool which is now working without problems, I just need to consolidate old snapshots.

Environment:

Kubernetes 1.19 with CSI driver v2.2.0 on VMs in 15 HW version. ESX 7.0.1 16850804, vcenter 7.0.1 17005016. vSAN stretched cluster with 2 locations and 6 hosts.

Which information I should share for better view to my issue?

Thank you! 

 

Reply
0 Kudos
8 Replies
Tom31415
Enthusiast
Enthusiast

But I found that there is a lot of .lck it is probably lock is`n it? But look at this output:

 

[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] ls -la 
total 10248
drwx------    1 root     root          4060 Jun  2 07:17 .
drwx------    1 root     root         16940 Jun  2 00:01 ..
-rw-------    1 root     root             0 Jun  2 06:58 .002cb760-5bbb-9101-2483-0cc47aa8ddd6.lck
-rw-------    1 root     root             0 Apr 21 14:10 .44328060-d729-0400-f29c-0cc47aa8ddd6.lck
-rw-------    1 root     root             0 May 31 23:59 .5f78b560-eea2-c40e-e388-0cc47aa8ddd6.lck
-rw-------    1 root     root             0 May 18 13:30 .63c1a360-b9fc-ff28-222a-0cc47aa8ddc8.lck
-rw-------    1 root     root             0 May 30 23:57 .8626b460-8491-98ac-2203-0cc47aa8ddc8.lck
-rw-------    1 root     root             0 Jun  2 07:17 .8730b760-c1ac-2ead-d4c4-0cc47aa8ddd6.lck
-rw-------    1 root     root             0 May 31 13:42 .d5e7b460-05a7-35f7-5a20-0cc47aa8ddd6.lck
-rw-------    1 root     root             0 May 31 07:01 .e389b460-2c34-9fcf-adf7-0cc47aa8ddc8.lck
-rw-------    1 root     root             0 Jun  1 23:59 .e8c9b660-cdb0-e1b1-2bca-0cc47aa8ddd6.lck
-rw-------    1 root     root        135168 Jun  2 07:05 1d07f42fa0294be3b6847f3c1448bcbf-000001-3a365258c9263af2.vmfd
-rw-------    1 root     root           607 Jun  2 07:17 1d07f42fa0294be3b6847f3c1448bcbf-000001.vmdk
-rw-------    1 root     root        135168 May 30 23:57 1d07f42fa0294be3b6847f3c1448bcbf-000002-b6ffcbdac3bba6ac.vmfd
-rw-------    1 root     root           630 Jun  2 00:29 1d07f42fa0294be3b6847f3c1448bcbf-000002.vmdk
-rw-------    1 root     root        135168 May 31 09:29 1d07f42fa0294be3b6847f3c1448bcbf-000003-ae608fe9d085bf9c.vmfd
-rw-------    1 root     root           630 May 31 19:06 1d07f42fa0294be3b6847f3c1448bcbf-000003.vmdk
-rw-------    1 root     root        135168 Jun  1 23:59 1d07f42fa0294be3b6847f3c1448bcbf-000004-1142f80f281e4425.vmfd
-rw-------    1 root     root           607 Jun  2 06:58 1d07f42fa0294be3b6847f3c1448bcbf-000004.vmdk
-rw-------    1 root     root        135168 Jun  2 00:29 1d07f42fa0294be3b6847f3c1448bcbf-000005-f84464ad880ca413.vmfd
-rw-------    1 root     root           623 Jun  2 00:29 1d07f42fa0294be3b6847f3c1448bcbf-000005.vmdk
-rw-------    1 root     root        135168 May 31 19:12 1d07f42fa0294be3b6847f3c1448bcbf-000006-5ae8aefb7d359fee.vmfd
-rw-------    1 root     root           630 May 31 23:59 1d07f42fa0294be3b6847f3c1448bcbf-000006.vmdk
-rw-------    1 root     root        135168 Jun  1 12:14 1d07f42fa0294be3b6847f3c1448bcbf-000007-e2a1f8a84504dbf0.vmfd
-rw-------    1 root     root           630 Jun  1 23:59 1d07f42fa0294be3b6847f3c1448bcbf-000007.vmdk
-rw-------    1 root     root        135168 Jun  2 07:17 1d07f42fa0294be3b6847f3c1448bcbf-000008-2dbdcd30f2eec255.vmfd
-rw-------    1 root     root           471 Jun  2 07:17 1d07f42fa0294be3b6847f3c1448bcbf-000008.vmdk
-rw-------    1 root     root        135168 May 22 00:31 1d07f42fa0294be3b6847f3c1448bcbf-899b8a5a353e20af.vmfd
-rw-------    1 root     root           967 May 31 19:07 1d07f42fa0294be3b6847f3c1448bcbf.vmdk
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000001.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000001.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000001.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.042716953903436661 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000002.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000002.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000002.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.036854330450296402 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000003.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000003.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000003.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.035995949059724808 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000004.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000004.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000004.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.037847887724637985 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000005.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000005.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000005.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.040791366249322891 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000006.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000006.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000006.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.036646779626607895 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000007.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000007.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000007.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.041259091347455978 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] 
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf-000008.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf-000008.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf-000008.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.037505663931369781 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] 
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] vmfsfilelockinfo -p 1d07f42fa0294be3b6847f3
c1448bcbf.vmdk
vmfsfilelockinfo Version 2.0
Looking for lock owners on "1d07f42fa0294be3b6847f3c1448bcbf.vmdk"
"1d07f42fa0294be3b6847f3c1448bcbf.vmdk" is not locked by any ESX host and is Free
Total time taken : 0.036234911531209946 seconds.
[root@esx1:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100]
Reply
0 Kudos
Tom31415
Enthusiast
Enthusiast

And I can see that the lck files is holding my k8s-node1 VM.

 

[root@esx4:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100] lsof | grep -E '(Cartel|82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096)'
Cartel    |      World name     |        Type         |   fd   |  Description     
25746959    grep                  cwd                        -1   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100
25746958    lsof                  cwd                        -1   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100
25746417    sh                    cwd                        -1   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100
15599672    vmx                   FILE                      105   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.8730b760-c1ac-2ead-d4c4-0cc47aa8ddd6.lck
15599672    vmx                   FILE                      131   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.002cb760-5bbb-9101-2483-0cc47aa8ddd6.lck
15599672    vmx                   FILE                      133   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.e8c9b660-cdb0-e1b1-2bca-0cc47aa8ddd6.lck
15599672    vmx                   FILE                      135   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.5f78b560-eea2-c40e-e388-0cc47aa8ddd6.lck
15599672    vmx                   FILE                      137   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.d5e7b460-05a7-35f7-5a20-0cc47aa8ddd6.lck
15599672    vmx                   FILE                      139   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.e389b460-2c34-9fcf-adf7-0cc47aa8ddc8.lck
15599672    vmx                   FILE                      141   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.8626b460-8491-98ac-2203-0cc47aa8ddc8.lck
15599672    vmx                   FILE                      143   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.63c1a360-b9fc-ff28-222a-0cc47aa8ddc8.lck
15599672    vmx                   FILE                      145   /vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100/.44328060-d729-0400-f29c-0cc47aa8ddd6.lck
[root@esx4:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100]
Reply
0 Kudos
TheBobkin
Champion
Champion

@Tom31415, That is normal.

With vSAN Objects you have to actually query the .lck itself for locker/free info e.g.:

# vmfsfilelockinfo -p .002cb760-5bbb-9101-2483-0cc47aa8ddd6.lck

or a loop:

# for x in `ls .*.lck`;do vmfsfilelockinfo -p $x;done

 

Tom31415
Enthusiast
Enthusiast

Hi, @TheBobkin,

Thank you! You are right, I can see that it is my esx2.brm host and it is k8s-node1 VM running on it. Do you have some idea how to free up these locks? Will be power off of VM enough or is there some other way without VM restart?

 

vmfsfilelockinfo Version 2.0
Looking for lock owners on ".002cb760-5bbb-9101-2483-0cc47aa8ddd6.lck"
".002cb760-5bbb-9101-2483-0cc47aa8ddd6.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.38427174463868141 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".44328060-d729-0400-f29c-0cc47aa8ddd6.lck"
".44328060-d729-0400-f29c-0cc47aa8ddd6.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.37261660397052765 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".5f78b560-eea2-c40e-e388-0cc47aa8ddd6.lck"
".5f78b560-eea2-c40e-e388-0cc47aa8ddd6.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.371800746768713 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".63c1a360-b9fc-ff28-222a-0cc47aa8ddc8.lck"
".63c1a360-b9fc-ff28-222a-0cc47aa8ddc8.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.3574189804494381 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".8626b460-8491-98ac-2203-0cc47aa8ddc8.lck"
".8626b460-8491-98ac-2203-0cc47aa8ddc8.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.36978151649236679 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".8730b760-c1ac-2ead-d4c4-0cc47aa8ddd6.lck"
".8730b760-c1ac-2ead-d4c4-0cc47aa8ddd6.lck" is locked in Exclusive mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Exclusive
Total time taken : 0.37831085547804832 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".d5e7b460-05a7-35f7-5a20-0cc47aa8ddd6.lck"
".d5e7b460-05a7-35f7-5a20-0cc47aa8ddd6.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.37819376587867737 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".e389b460-2c34-9fcf-adf7-0cc47aa8ddc8.lck"
".e389b460-2c34-9fcf-adf7-0cc47aa8ddc8.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.41381392255425453 seconds.
vmfsfilelockinfo Version 2.0
Looking for lock owners on ".e8c9b660-cdb0-e1b1-2bca-0cc47aa8ddd6.lck"
".e8c9b660-cdb0-e1b1-2bca-0cc47aa8ddd6.lck" is locked in Read-Only mode by host having mac address ['0c:c4:7a:a8:dd:d6']
Trying to make use of Fault Domain Manager
----------------------------------------------------------------------
Found 6 ESX hosts using Fault Domain Manager.
----------------------------------------------------------------------
Searching on Host esx2.kl.xxxxxxx
Searching on Host esx3.kl.xxxxxxx
Searching on Host esx2.brm.xxxxxxx
    MAC Address : 0c:c4:7a:a8:dd:d6


Host owning the lock on file is esx2.brm.xxxxxxx, lockMode : Read-Only
Total time taken : 0.35222738981246948 seconds.
[root@esx4:/vmfs/volumes/vsan:52e071236c8cc082-86e2156602f6684c/82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_0096/_0100]

 

Thank you!

Reply
0 Kudos
TheBobkin
Champion
Champion

@Tom31415, Again, it is completely normal for the host that a VM is running on is locking the vmdks - this is a basic protection mechanism.


However, this doesn't mean that there isn't something else running on that host (e.g. backup-proxy VM) or physical backup server commands via the ESXi host also locking the files - one very basic check I do with such VMs is (if possible) migrate it to another host and see do some locks on some files still say the original host is locking them.


Can you check if there is anything other than the expected files/handles accessing these in case they are still hot-added to backup proxy or some such (run on node esx2.brm or wherever it resides now):

# lsof | grep 1d07f42fa0294be3b6847f3c1448bcbf

 

Powering off the VM is unlikely to stop whatever is locking it, if it is a physical backup appliance then rebooting the appliance may break the lock but sometimes (e.g. if there is some command previously issue by backup appliance (or host command resultant of this)) the host itself will need to be rebooted.

Reply
0 Kudos
Tom31415
Enthusiast
Enthusiast

Ok, thanks, these lck files were only single clue which I had. I tried grep lsof output and did not find any other process which is holding my disks so I assume that my problem is not cause by locks. And it is returning me to beginning of my problem where I have no idea why is esx not consolidating my disks during Consolidating triggered by Consolidation option nor by Delete All Snapshots.

The single option which I did not try is to shutdown cluster and do snapshots with deleting all snapshots. I am gonna try it.

 

govc output of one of my broken disk:

{
                "Key": 2011,
                "DeviceInfo": {
                  "Label": "Hard disk 7",
                  "Summary": "10,485,760 KB"
                },
                "Backing": {
                  "FileName": "[vsanDatastore] 82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_005a/0a87082116784649877aad03b411b92e-000002.vmdk",
                  "Datastore": {
                    "Type": "Datastore",
                    "Value": "datastore-193"
                  },
                  "BackingObjectId": "231bb860-8ebf-1b6e-dd21-0cc47aa8ddd6",
                  "DiskMode": "persistent",
                  "Split": false,
                  "WriteThrough": false,
                  "ThinProvisioned": true,
                  "EagerlyScrub": false,
                  "Uuid": "6000C292-c6d1-36e8-690d-c9bf153ed680",
                  "ContentId": "297d4ce453058ea1eb5ba36f08aceae3",
                  "ChangeId": "",
                  "Parent": {
                    "FileName": "[vsanDatastore] 82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_005a/0a87082116784649877aad03b411b92e-000001.vmdk",
                    "Datastore": {
                      "Type": "Datastore",
                      "Value": "datastore-193"
                    },
                    "BackingObjectId": "93c9b660-963e-1b85-95ff-0cc47aa8ddd6",
                    "DiskMode": "persistent",
                    "Split": null,
                    "WriteThrough": null,
                    "ThinProvisioned": true,
                    "EagerlyScrub": false,
                    "Uuid": "6000C292-c6d1-36e8-690d-c9bf153ed680",
                    "ContentId": "608ce38ebdbf2bb0a986b794d0f7f75a",
                    "ChangeId": "",
                    "Parent": {
                      "FileName": "[vsanDatastore] 82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_005a/0a87082116784649877aad03b411b92e-000005.vmdk",
                      "Datastore": {
                        "Type": "Datastore",
                        "Value": "datastore-193"
                      },
                      "BackingObjectId": "0d78b560-0171-457c-745e-0cc47aa8ddd6",
                      "DiskMode": "persistent",
                      "Split": null,
                      "WriteThrough": null,
                      "ThinProvisioned": true,
                      "EagerlyScrub": false,
                      "Uuid": "6000C292-c6d1-36e8-690d-c9bf153ed680",
                      "ContentId": "1c405a89ab8a7fed5a631a580d7b17d8",
                      "ChangeId": "",
                      "Parent": {
                        "FileName": "[vsanDatastore] 82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_005a/0a87082116784649877aad03b411b92e-000007.vmdk",
                        "Datastore": {
                          "Type": "Datastore",
                          "Value": "datastore-193"
                        },
                        "BackingObjectId": "59c1a360-bcf5-3ade-5dc8-0cc47aa8ddd6",
                        "DiskMode": "persistent",
                        "Split": null,
                        "WriteThrough": null,
                        "ThinProvisioned": true,
                        "EagerlyScrub": false,
                        "Uuid": "6000C292-c6d1-36e8-690d-c9bf153ed680",
                        "ContentId": "0eb0bbf38722db3e496a96d2dc573c9e",
                        "ChangeId": "",
                        "Parent": {
                          "FileName": "[vsanDatastore] 82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_005a/0a87082116784649877aad03b411b92e-000003.vmdk",
                          "Datastore": {
                            "Type": "Datastore",
                            "Value": "datastore-193"
                          },
                          "BackingObjectId": "529d7960-e341-f71c-3738-0cc47aa8ddc8",
                          "DiskMode": "persistent",
                          "Split": null,
                          "WriteThrough": null,
                          "ThinProvisioned": true,
                          "EagerlyScrub": false,
                          "Uuid": "6000C292-c6d1-36e8-690d-c9bf153ed680",
                          "ContentId": "e7ba8d22a80bbe92a11f29319ca1cba3",
                          "ChangeId": "",
                          "Parent": {
                            "FileName": "[vsanDatastore] 82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_005a/0a87082116784649877aad03b411b92e-000004.vmdk",
                            "Datastore": {
                              "Type": "Datastore",
                              "Value": "datastore-193"
                            },
                            "BackingObjectId": "366e6760-b139-77f3-ceb3-0cc47aa8ddc8",
                            "DiskMode": "persistent",
                            "Split": null,
                            "WriteThrough": null,
                            "ThinProvisioned": true,
                            "EagerlyScrub": false,
                            "Uuid": "6000C292-c6d1-36e8-690d-c9bf153ed680",
                            "ContentId": "0b275b5b8bac2541c7225361c7d0fc58",
                            "ChangeId": "",
                            "Parent": {
                              "FileName": "[vsanDatastore] 82b4bb5f-d24b-a89f-7d02-0cc47aa8de6e/_005a/0a87082116784649877aad03b411b92e.vmdk",
                              "Datastore": {
                                "Type": "Datastore",
                                "Value": "datastore-193"
                              },
                              "BackingObjectId": "20826160-506a-0ce3-3a0a-0cc47aa8de6e",
                              "DiskMode": "persistent",
                              "Split": null,
                              "WriteThrough": null,
                              "ThinProvisioned": true,
                              "EagerlyScrub": false,
                              "Uuid": "6000C292-c6d1-36e8-690d-c9bf153ed680",
                              "ContentId": "b02ad8b8fa796770e1b4257cdecee509",
                              "ChangeId": "",
                              "Parent": null,
                              "DeltaDiskFormat": "",
                              "DigestEnabled": false,
                              "DeltaGrainSize": 0,
                              "DeltaDiskFormatVariant": "",
                              "Sharing": "",
                              "KeyId": null
                            },
                            "DeltaDiskFormat": "redoLogFormat",
                            "DigestEnabled": false,
                            "DeltaGrainSize": 0,
                            "DeltaDiskFormatVariant": "vsanSparseVariant",
                            "Sharing": "",
                            "KeyId": null
                          },
                          "DeltaDiskFormat": "redoLogFormat",
                          "DigestEnabled": false,
                          "DeltaGrainSize": 0,
                          "DeltaDiskFormatVariant": "vsanSparseVariant",
                          "Sharing": "",
                          "KeyId": null
                        },
                        "DeltaDiskFormat": "redoLogFormat",
                        "DigestEnabled": false,
                        "DeltaGrainSize": 0,
                        "DeltaDiskFormatVariant": "vsanSparseVariant",
                        "Sharing": "",
                        "KeyId": null
                      },
                      "DeltaDiskFormat": "redoLogFormat",
                      "DigestEnabled": false,
                      "DeltaGrainSize": 0,
                      "DeltaDiskFormatVariant": "vsanSparseVariant",
                      "Sharing": "",
                      "KeyId": null
                    },
                    "DeltaDiskFormat": "redoLogFormat",
                    "DigestEnabled": false,
                    "DeltaGrainSize": 0,
                    "DeltaDiskFormatVariant": "vsanSparseVariant",
                    "Sharing": "",
                    "KeyId": null
                  },
                  "DeltaDiskFormat": "redoLogFormat",
                  "DigestEnabled": false,
                  "DeltaGrainSize": 0,
                  "DeltaDiskFormatVariant": "vsanSparseVariant",
                  "Sharing": "sharingNone",
                  "KeyId": null
                },
                "Connectable": null,
                "SlotInfo": null,
                "ControllerKey": 1000,
                "UnitNumber": 11,
                "CapacityInKB": 10485760,
                "CapacityInBytes": 10737418240,
                "Shares": {
                  "Shares": 1000,
                  "Level": "normal"
                },
                "StorageIOAllocation": {
                  "Limit": -1,
                  "Shares": {
                    "Shares": 1000,
                    "Level": "normal"
                  },
                  "Reservation": 0
                },
                "DiskObjectId": "49-2011",
                "VFlashCacheConfigInfo": null,
                "Iofilter": null,
                "VDiskId": {
                  "Id": "cd8be968-e88f-4bc5-b848-474dfdf0bc44"
                },
                "NativeUnmanagedLinkedClone": false
              }

 

Reply
0 Kudos
Tom31415
Enthusiast
Enthusiast

I tried creating and deleting all snapshots from my vms when it was powered off and it did not help. Snapshots removing just ends well but delta files are still there. And I did not find any errors. 

Reply
0 Kudos
Tom31415
Enthusiast
Enthusiast

I have no idea why is my consolidation process and all snapshots deleting process ending sucessfully but my delta files are still there without touch. I have no error about it but deltas are still same.

Reply
0 Kudos