I have 3x identical Supermicro machines each with 8c/16t, 64GB ram, 512GB ESXi Drive, 1TB NVME Cache Drive, 6.4TB NVME Capacity drive. The is a very simple setup for my homelab on 1 switch with 2 VLANs.
I just installed esxi 7 on all three machines with the management interface in a vlan. It pulls the proper IP address from the static DHCP server.
I then installed VCSA 7 from the UI installer on windows.
I turned on vMotion and vSAN, I added the 2x other hosts to make it so all three are online. I set static address on a new vnic for vMotion and vSAN.
After the system was running fairly stable for a few hours I went to create a new vm and the VCSA interface seems slow.
As you can see it looks like there was an attempt to migrate the VCSA vm and failed but I don't know why.
Please let me know where to pull more logs that would help to troubleshoot this issue.
Trying to fill out more details as I come across them.
Host 1 message: vSphere HA agent for this host has an error: The vSphere HA agent is not reachable from vCenter Server
All hosts have the saem looking network settings:
vmk0 is listed for management, it's on my main vlan that is for all servers
vmk1 is liked for vMotion and vSAN it's on a storage vlan (I think I figured out how to put a DSwitch in a vlan).
While the DSwitch-vSAN has DHCP, all hosts are hardcoded to match the management vlan number scheme.
I realized that the vSAN and vMOTION vmk1 was set to 1500MTU and my 10G switch supports 9000 MTU jumbo frame. My VCSA freezes for 5-10 minutes when I went to make the change by editing vmk1 to make it 9000 MTU. VCSA and all of my traffic is on vmk0 so why would VCSA hang for so long?
I was recently running Proxmox with Ceph for a similar Hyper-converged solution on this network for learning purposes and I never ran into a problem like this. With all drives being NVME the solution was really really fast, I'm just not sure what I have wrong with vSphere to have these kinds of troubles after just installing it.