No more VM in my VSAN and already 130 G used???

[ekrejci]

Hi,

I've deployed a tiny vSan yesterday. Only 3 hosts with 100G of data each.

After that I've deployed a single VM of 65G. this went fine as well. I tried to change the storage capability for this vm from Raid1 to Raid0 (Number of failures to tolerate -> 0) I received an error at my VM. the vm was shown as "disappeared" from the Web Client. I tried to browse the vSan in the VM's folder ad didn't find its vmdk!

So I removed everything manually (the vm from the inventory -> no other choice) and the VM's folder on the vSan datastore.

After a period of time and activity on the hosts disks, I'm having the following storage capacity reported:

df -h

Filesystem   Size Used Available Use% Mounted on

VMFS-5       9.8G 1.9G      7.9G  19% /vmfs/volumes/Local_22

vfat         4.0G 36.4M      4.0G   1% /vmfs/volumes/5326c8d8-74ce1025-ec48-005056b928f3

vfat       249.7M 182.3M     67.4M 73% /vmfs/volumes/4f784977-d5d707dc-c2c0-9b8e5e026067

vfat       249.7M 166.8M     82.9M 67% /vmfs/volumes/fb7ac170-5e14d697-37ca-8920eb265df4

vfat       285.8M 192.6M     93.2M  67% /vmfs/volumes/5326c8d1-58ce96e2-e0bb-005056b928f3

vsan       299.2G 130.3G    169.0G 44% /vmfs/volumes/vsanDatastore

so why if this vSan using 130.3G. if I 'ls' it it is totally empty:

ls -lah /vmfs/volumes/vsanDatastore/

total 0

drwxr-xr-x    1 root root         512 Mar 18 07:22 .

drwxr-xr-x    1 root root         512 Mar 18 07:22 ..

~ #

I tried to disable the VSAN and re-enable it without any change.

Someone know how I can check what's into it?

Many thanks

Eric

[arvinnd]

>>so why if this vSan using 130.3G. if I 'ls' it it is totally empty:

This is due to the fact that you created a VM of of 65GB with Number of Host Failures to tolerate as 1 and for n failures tolerated, n+1 copies of the virtual machine object are created.

[depping]

on top of that, when you reconfigure it ... it will spin up a new "storage instance" so you are probably consuming the capacity * 3 temporarily.

Just tested it, this only applies to the scenario when changing striped.

[depping]

Also, are you running supported hardware? (vmwa.re/vsanhcl)

[ekrejci]

Hi,

the N+1 copy I understand. what I don't is why even after having removed the VM the space has not been freed?

Meanwhile I continued to do some investigations and manipulations.

I came with this results:

turning VSAN off than back ON in auto did not freed the space.

changing the VSAN from auto to manual and then manually removing every disk and putting them back again did free the space. I have now an empty vsan datastore that I can fully use.

it seams that some "ghost objects" where left behind after the error I had with my VM.

does a tools can show if orphaned objects exist on the vsan datastore?

Eric

[ekrejci]

Hi Duncan,

I have to admit that I'm not completely in the HCL, this might has caused the initial error I had with my VM. but I'm in a lab to see the VSAN functionalities, and of course deploying VSAN in a POC and Production state, I would stick to the HCL

but nevertheless I'm wondering about orphaned objects now that it seems I've had some. and the way to detect them.

Eric

[depping]

which disk controller are you using?

[depping]

ekrejci wrote:

but nevertheless I'm wondering about orphaned objects now that it seems I've had some. and the way to detect them.

Eric

Looking at that right now, but I see no option to identify orphaned objects or components quickly

[ekrejci]

a PERC 6i on a Dell PE R805

Eric

[arvinnd · Accepted solution]

RVC tool can be of help in getting vsan vm object information.  Following link has detailed information. I believe in the first release finding and deleting orphan objects is not available but it seems necessary.

Manage VSAN with RVC Part 3 – Object Management | Virten.net

I have not tested but using following commands we can identify orphan objects.

vsan.vm_object_info

vsan.cmmds_find

vsan.disk_object_info

Thanks

Arvind

[ekrejci]

Hi Arvind,

Thanks for these precious information and link

The virten.net part of vSAN at RVC is a very helpful guide to the whole think. I've also found this VMware doc:

VMware Virtual SAN - Quick Monitoring & Troubleshooting Reference Guide

Now I don't have any more orphaned object, I will try to reproduce it and then use the commands in RVC.

Do you also think that some very, very useful perf info that can be gathered with the 'vsan.observer' will be integrated directly in the Web client someday?

Anyway thank you guys for your help.

Sincerely

Eric