Hi all--- a few more notes.
tl;dr Learn esxtop. Assigning more resources to vm's can result in lowered performance.
I *finally* pinned down what seemed to be the last "stutter" problem. To put it briefly, running 4 vm's with 8 vcpu's each on a 16core /32 thread CPU was a bad idea. Under load, I discovered up to 50% READY state in esxtop which corresponded to massive stutter (should be < 5%; this is a measure of how often the vm was ready to execute but forced to wait due to lack of host resources). I'm theorizing that the win10 vm's didn't realize (weren't told) that those "8 vcpus" were really "4 cores/ 8 thread." REDUCING the number of vcpu's on each vm from 8 down to 4 massively IMPROVED performance, with ready state < 0.5%.. It also fixed the latencymon testing (now the vm's pass, yay!).
vm's with GPU's all have RAM reserved. If you're running in NUMA mode, you probably want all that RAM in one NUMA domain. The order in which you poweron your vm's, and how much ram they have, can affect esxi's ability to do this. Again, more can be less. I haven't investigated UMA vs. NUMA comprehensively.
Moved back to the asrock x399 Professional Gaming MB, works. Again, only 2 MB USB pass through successfully. There seems to be a non-vmware driver available for the 10G NIC (aquantia).
All vm's moved to 6.5 NVME controllers successfully.
