All Flash VSAN Physical Disk re-balance issue - There is no more space for virtual Disk 'vm-name.vmdk'

[vHaridas]

Hi,

Need expert advice on all flash VSAN; so I came back to VMware community after long time.

This is 6 Node All Flash VSAN cluster with deduplication and compression enabled. 

Please let me know if anyone has any suggestion or questions? i can share required info.

My End user is not at all happy with this and he want to remove all flash vsan cluster and go back to physical server for his big sql VMs, really disappointing experience.

am unable to power on VM, though there is 27 TB free space available on VSAN datastore.

Affected VM has vmdk of 13 TB. ( 1 disk of 100 GB and another disk of 13 TB )

After looking through Physical disk utilization I can see there is big difference between physical utilization across hosts. some disks are utilized 9% and other's are hitting 90%

One of the host's all Physical Disk's are 89% to 95% utilized which has object of affected VM.

Error - There is no more space for virtual Disk 'vm-name.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, clicking Retry. Click Cancel to terminate this sessions.

Here is what I did -

1. using RVC, started proactive disk re-balance

2. Created ticket with VMware support to look in to cluster. Support suggested me to continue disk rebalncing task.

3. Next day after 14 hours, I stopped proactive rebalance as I had to upgrade vcenter which was not working correctly.  I had ask if I can do this and support was ok with stopping proactive disk rebalance and upgrading vCenter and then starting disk rebalance.

4. After vcenter upgrade, again I started proactive disk rebalance using rvc.

5. rebalance ran for few hours and it stopped automatically without doing anything.

6. Then again I started proactive rebalance using rvc; it ran for 24 hours but nothing happened. all physical disk utilization was as it is.

7. Then VMware support asked me to put host in Maintenance Mode which has fully utilized disk and select all data evacuation  so that affected VMs data will be migrated across all other disks on other hosts.

Even after 24 hours, Host Maintenance Mode process is still in progress. it is at 76% since last 12 hours.

VMware SR#17591698210

VMware support is also not able to do much except checking objects counts, stats and currently running vsan task.

This VMs total size is 13 TB, so if anything I do is going to take lot of time.

Few more option I think of -

1. Clone VM

2. Storage vMotion to other Shared Storage (SAN)

3. Replicate VM to other SAN storage

4. Delete VM or affected vmdk and ask user to create new one - if I do this, user will never allow me to use all flash VSAN   believe me.

Host Configuration (per server) -  6 node VSAN Cluster

Cisco Systems Inc UCSC-C240-M4SX

2 x Intel Xeon CPU E5-2680 v3 @ 2.50GHz

512 GB RAM

LSI MegaRAID SAS 3108

Disk Group-1 - 1 SSD Cache and 5 SSD Capacity Disk

Disk Group-2 - 1 SSD Cache and 4 SSD Capacity Disk.

This cluster has only 4 Vms but with large RAM and vmdk with size up to 13 TB

Thanks,

Haridas

[AdWi]

Hi,

What storage policy are you using for this VM?

Have you tried to change stripe width for this particular VM?

[EvgenySemenenko]

Hello Haridas,

I'm experiencing similar issue on VSAN 6.6.1. I must admit proactive re-balance works  better for me than you describe. I'm on stretched cluster with raid5 in nested fault domains with dedup&comp ON. Testing 1 host rebuild brings down the whole cluster because the host which is being rebuilt is running out of disk space, while all others operate at their current levels. Basically from my observations I can say that proactive re-balance process is not fast enough to free up the space for resync process, which is aggressively copying the data to rebuild objects on failed host. So far I have found no workaround, except for temporary fix - put the host to MM again. I also have an SR opened but so far haven't seen a resolution.

Any info on this would be appreciated. Looks like a very nasty bug to me. 

p.s. coming back to your original problem, individual disk/host levels are much better kept in VSAN 6.6, I see more even distribution of data than you describe. I'd consider an upgrade if it is feasible.

Regards

Evgeny

[hkg2581]

Hello Haridas,

I just came across your post . As discussed on our conversation over the SR , we didnot have enough free space on the vSAN datastore to re-apply Erasure coding policy on the VM objects . Please let me know if you still have trouble post applying the Erasure coding policy Raid-5/6 .

vHaridas The Original issue was because of the size of the components , vSAN could not find a host which can move these components as they would fill up the destination DG .

I have tried to explain object and component placement here Understanding objects and component creation and placement in vSAN feel free to share any feedback.

Regards,

Hareesh K G

[vHaridas · Accepted solution]

Hi Everyone,

Thanks Everyone for responding. I apologies for not responding earlier.

Hareesh G from VMware support helped me to resolve this issue.

Overall my observation is vSAN is not able to handle big VMDK's properly. It should provide errors or warnings if there is not enough resources available to satisfy big vmdk's.

It is not moving VM object to other hosts when there is enough free space available on other host or once host is comes online from Maintenance.

Hareesh came with a solution of disabling Vmkernel adapter of vSAN on affected host.

This forced vSAN to rebuild all components to other hosts.

With this out of space issue was resolved but still my challenges with this big disks are not finished.

Current Status -

I have applied new storage policy to all 6 VMs except 13.5 TB big vmdk.

New Policy - FTT=2,  Stripe per object=2, Raid 5/6.

Still, one set of VMDK object of this Vmdk is using 90% of physical disks capacity from only one of the host. while other host has free space.

Next plan - am going to migrate this big Vmdk to other temporary SAN datastore, and then move back to vSAN with new raid 6 vm policy.

This will split vmdk objects across all hosts.

With above workaround, my all hosts disks would be equally utilized and objects would be placed properly on all physical disks.

But next time when I will have long downtime on any of the host then vSAN will rebuild that Hosts object to other host in cluster.

again, only one of the hosts disks will get fully utilized due to this.

So what I want is, when failed/host comes out of Maintenances then vSAN should do rebalance of all objects and make sure all disks are equally used.

I tried to run proactive rebalance multiple times but it is of no use when you have big size VMDKs.

Hareesh, you have explained VM object placement very well but as I said big VMDKs are difficult to manage on vSAN unless we plan for big size physical Disk and more physical hosts.

I have asked my team to not assign big Vmdk to vSAN Vms and trying to limit Vmdk size to 2 or 4 TB.

2 or 4 TB Disk is just a number, it matters how much is the size of physical disks and number of physical disks.

small size Vmdk will help to split Vmdk size and place vmdk objects on multiple hosts across all hosts.

Thanks,

Haridas