Trying to fill out more details as I come across them.
Host 1 message: vSphere HA agent for this host has an error: The vSphere HA agent is not reachable from vCenter Server
All hosts have the saem looking network settings:
vmk0 is listed for management, it's on my main vlan that is for all servers
vmk1 is liked for vMotion and vSAN it's on a storage vlan (I think I figured out how to put a DSwitch in a vlan).
While the DSwitch-vSAN has DHCP, all hosts are hardcoded to match the management vlan number scheme.
I realized that the vSAN and vMOTION vmk1 was set to 1500MTU and my 10G switch supports 9000 MTU jumbo frame. My VCSA freezes for 5-10 minutes when I went to make the change by editing vmk1 to make it 9000 MTU. VCSA and all of my traffic is on vmk0 so why would VCSA hang for so long?
I was recently running Proxmox with Ceph for a similar Hyper-converged solution on this network for learning purposes and I never ran into a problem like this. With all drives being NVME the solution was really really fast, I'm just not sure what I have wrong with vSphere to have these kinds of troubles after just installing it.