Here are my thoughts. They're worth exactly what you paid for them. 🙂
1. Has anyone seen any documentation on format of Fusion virtual disks? Is it just a simple duplicate of what's on the raw metal or is there structure in it. I am interested in such things as Fusion snapshot vs APFS snapshot, deleted space recovery, etc.
Logically the virtual disk is a image of a physical disk - what you'd see starting at block 0 of a physical disk. Physically there are different configurations of the files that make up the virtual disk depending on how you've configured it.
Fusion itself doesn't document how a virtual disk is built other than describing how to configure a virtual disk to be monolithic (all contained in one file) or split into multiple pieces - or whether it is fully or "thinly" allocated. I've been trying to search for documentation on the "internal" guts of how a virtual disk is configured. @a_p_ who's our resident guru on VMDKs seems to have the best handle on it, and maybe he can point you to any VMware documentation that goes into more detail on exactly how it's built. There are several references to Virtual Disk 5.0 format tech note, in VMware documentation, but it does not seem to be able to be found anywhere.
VMware snapshots are built as a series of "delta" files that use internal pointers in the vmdk data structure to give the illusion of a full copy of the disk but only needing to keep track of what "blocks" have changed. When you create a vmdk snapshot, a new set of files are created corresponding to the files that you have configured for the current vmdk file. This series of user-land visible pointers in these files and extra files are what makes vmdk snapshots brittle - it doesn't take much to interrupt a write to maintain the chains of pointers that describe the snapshot (and it gets worse when you take snapshot on top of snapshot on top of snapshot - from a performance and stability standpoint).
APFS snapshots are a totally different animal and taken at the APFS volume level by the operating system. Fusion has no idea how to orchestrate them. The concept is similar, but IMO much better implemented. Key things about APFS file operations:
- Files are made up of multi-block extents. Each extent has a reference count. That is, it knows how many things are pointing to it.
- An APFS snapshot is virtually instantaneous for the file system at the OS level. It makes a copy of the current file system metadata (which makes it look like the state of the file system at that point in time). All extents referenced get their reference count updated by 1.
- Every write to the main volume is a copy on write to an extent (set of blocks). It doesn't write back to the same block, it writes data to a new extent on disk. The old one gets its reference count decremented.
- When an extent has its reference count decremented to zero, the extent's space can be returned to the disk pool. As long as it still has a non-zero reference count, the block stays around because it's pointed to you by a snapshot.
- APFS snapshots are currently read-only.
All of this makes APFS snapshots much more reliable than VMware snapshots, IMO. The VMware implementation, while portable, makes it brittle IMO. It needs a sequence of writes to multiple files that need to be completed "at once" to maintain the snapshot. One burp that doesn't write (or flush) all of those changes back to disk, and the vmdk file can require some form of recovery or break entirely.
2. Any thoughts on using a COW (copy on write) file system such as Btrfs or ZFS on a VM under Fusion?
My opinion: COW file systems in the guest are much better for recovery and stability in the guest. They have the side effect of driving up host disk consumption for the VMDK files because they, well, copy on write to a new block. I prefer Btrfs over ZFS, though, because ZFS has a habit of using memory as cache to improve its performance. Your preferences may be different.
3. Which Linux file system would give best performance? I suspect on a virtual disk a simpler file system such as ext4 might be a good choice, maybe even with journaling off to reduce writes. Does Fusion do TRIM on virtual disks or assume OS will handle it?
I don't think that TRIM is passed through to the underlying host operating system. Reclaiming space on a virtual disk that isn't fully allocated is not a pretty experience - it usually required defragging, writing zeros to unalllocated space, and then manually running vmware-vdiskmanager to "compact" the host files that make up the virtual disk.
4. Finally I know the APFS/HPFS tradeoffs for HDD vs SDD. Same for Fusion files?
Yes. Since vmdk files are implemented as operating system files, the general advice for where to use APFS vs HFS+ still remains. Use APFS for SSDs, Time Machine HDDs, and HDDs that contain mostly static information. Use HFS+ on all other HDDs, including HDDs used to store Fusion VMs.
Every
- Paul (Technogeezer)
Editor of the Unofficial Fusion Companion Guides
