Optimizing vmxnet3 for intrahost guest-to-guest communication?

[richard612]

Hi all, wondering if there's any guidance for optimizing vmxnet3 advanced settings when there's heavy VM-to-VM network I/O within a single host.  Given that the data is never "hitting the wire" I'm wondering if it would be useful to tweak things like interrupt moderation, RSS scaling, along with other Windows parameters such as congestion providers, autotuning levels, scaling heuristics, etc.

Edit: I'm looking to optimize throughput over latency.  I've seen the "tuning for latency sensitive workloads" VMware PDF already.

[golddiggie]

For network traffic that will be isolated to a single host server, you could create a bubble network and give all the VMs connections on that. Make it with no pNICs so it's never even touching a wire. In such a configuration, the traffic will be at bus speeds (basically, the fastest you can get).

Do these VMs need outside of the host network access? If they still need to get access to the rest of the network, or your LAN, you could use routing tables (IIRC) to specify any traffic going to the other VMs goes either via the bubble network IP or connections.

IMO, I would go for the simplest solution (K.I.S.S. principle) and not over-think things.

[richard612]

I've already given the VMs a second "internal" network with jumbo frames enabled and I've confirmed this network is being used for the inter-VM communication. 

After I get Veeam updated I'm going to play around with iPerf in these VMs and see what I can come up with.