LevaEng Tracker

Re: Trim/Unmap support on GuestOS using FALLOC_FL_PUNCH_HOLE on HostOS (Thick provisioned vmdk)

LevaEng — Tue, 26 Sep 2023 09:44:05 GMT

It is impossible for the host machine to know which data in the guest disk it is actually used and which is not
For a full explanation on why see my answer in the other thread

While it is true that the Trim command (that is a sata command) is mostly associated with ssd it also true that the SCSI variant UnMap (that is the same thing) exist for this exact purpose of virtualisation and disk-provisioning

Re: Trim/Unmap support on GuestOS using FALLOC_FL_PUNCH_HOLE on HostOS (Thick provisioned vmdk)

LevaEng — Sat, 09 Sep 2023 11:53:08 GMT

Hi

Thank you for you answers
I will try to expand and clarify why thick-storage is better in this case

As support in the wild I can't verify right now but I'm 99% sure KVM/QEMU have TRIM support on GUEST and propagate it to the HOST (I used it before) (docs: driver -> discard)

About why not thin-provisioned disk?

My architetture stack is:

VMs: Windows 11 w NTFS 64K cluster size
Host: Workstation 17 on Linux
NFS over dedicated 10GBe
NAS: ZFS dataset w 64K record size & deduplication

With this solution I get the best performances from my system and a really good disk saving
Why: I found 64K sector is the sweetspot for my usage and it give good balance so that after deduplication and compression all VM disk stay nicely and hot in the ARC and L2ARC (ram and nvme cache respectively)

The problem with thin-provisioning is that I have no control over the allocation, granularity and most important alignment of it's internal chunks
If I put a thin disk on top of ZFS I'm putting a de-facto sparse datastructure on on top of a CoW filesystem, this create a really big write-amplification and also completely nullify deduplication given the fact that there is no more "sector/chunk" alignment from the Windows-NTFS to the actual storage-file

Why ZFS as backing storage?

Deduplication
Transparent compression
Automatic 0% performance penalty snapshot

I didn't use iSCSI on top of ZVOL mostly for simplify sysadmin operation for other in the team

The absence of TRIM/Discard/UnMap mean that block are allocated but never freed up
Yes you could enter every single machine and execute a SDelete on each of them and then at the end of the stack ZFS will recognise and free-up that chunk, but this IMHO is a really ugly hack

Hope this explain better why TRIM/Discard/UnMap support in important of Filesystem supporting FALLOC_FL_PUNCH_HOLE

Trim/Unmap support on GuestOS using FALLOC_FL_PUNCH_HOLE on HostOS (Thick provisioned vmdk)

LevaEng — Thu, 27 Jul 2023 14:00:44 GMT

Hi

I'm spinning multiple VMs on top of a ZFS filesystem dataset

Each VM has its disk fully allocated (Thick Provisioned), but given that ZFS support native compression (and deduplication of data too) the actual on-disk-size of each vmdk file is only as big as the written data (or even 1 / N with dedup)
Generally speaking this is what should happen on any modern FileSystem than support sparse-file / Fallocate/ Hole-Punching
(This setup is better than sparse vmdk because it have superior performance generally and extra far better storage efficiency if combined with ZFS native compression and on-line data deduplication )

This work good and well while the GuestOS write data, but then when such data is deleted it is never freed-up on the Host side because the HostOS can never know (if not hinted) that this range of the vmdk file has been "released" by the upper GuestOS

For this, at least on linux based OSes there is the fallocate( FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE ) that can "punch hole" in the file and free the not used space (logical size of the file stay the same but physical on-disk size shrink as needed)

But when configuring a VM on Workstation 17 with:
- Thick Provisioned Disk (single file)
- I/O controller : LSI Logic SAS
- Disk Type : NVMe

The GuestOS (Windows11) report that the disk do not have Trim support
But Trim support in this case should be supported and then converted in a fallocate call on the vmdk file by the HyperVisor (VMWare Workstation)

How can I can enable this behaviour? I already tried adding to the .vmx configuration file:
- nvme0:0.virtualSSD = "TRUE"
- disk.scsiUnmapAllowed = "TRUE"

But GuestOS is still reporting no Trim support

Thanks,
Luca

Also posted here

Trim/Unmap support on GuestOS using FALLOC_FL_PUNCH_HOLE on HostOS (Thick provisioned vmdk)

LevaEng — Thu, 27 Jul 2023 14:04:30 GMT

Hi

I'm spinning multiple VMs on top of a ZFS filesystem dataset

Each VM has its disk fully allocated (Thick Provisioned), but given that ZFS support native compression (and deduplication of data too) the actual on-disk-size of each vmdk file is only as big as the written data (or even 1 / N with dedup)
Generally speaking this is what should happen on any modern FileSystem than support sparse-file / Fallocate/ Hole-Punching
(This setup is better than sparse vmdk because it have superior performance generally and extra far better storage efficiency if combined with ZFS native compression and on-line data deduplication )

This work good and well while the GuestOS write data, but then when such data is deleted it is never freed-up on the Host side because the HostOS can never know (if not hinted) that this range of the vmdk file has been "released" by the upper GuestOS

For this, at least on linux based OSes there is the fallocate( FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE ) that can "punch hole" in the file and free the not used space (logical size of the file stay the same but physical on-disk size shrink as needed)

But when configuring a VM on Workstation 17 with:
- Thick Provisioned Disk (single file)
- I/O controller : LSI Logic SAS
- Disk Type : NVMe

The GuestOS (Windows11) report that the disk do not have Trim support
But Trim support in this case should be supported and then converted in a fallocate call on the vmdk file by the HyperVisor (VMWare Workstation)

How can I can enable this behaviour? I already tried adding to the .vmx configuration file:
- nvme0:0.virtualSSD = "TRUE"
- disk.scsiUnmapAllowed = "TRUE"

But GuestOS is still reporting no Trim support

Thanks,
Luca

Also posted here