VMware {code} Community
AndrewSK
Contributor
Contributor

VM Guests intercommunication api and performance

Hi all!

We need a fast method of communication between VM running on the same host/hypervisor (ESX(i) product line, guest os - windows xp-2003). Ideally, it's performance should be comparable with LPC on the target os (xp-2003).

I've found 2 sets of api

  1. Legacy VMCI api

  2. VMCI sockets.

Legacy api has an options of shared memory and datagram sockets - it is somehow fits our needs in terms of performance. It is not easy to design fast sync primitives with this, but possible. Socket api, on other side, give more generic api, but whats in terms of performance? I've found a couple of messages, where was stated about 10K ping-pong messages per second and for about 6 gbit (max!!!

0 Kudos
2 Replies
svaerke
VMware Employee
VMware Employee

Hi,

To answer your questions:

1) The legacy VMCI API is already deprecated and was never released for ESX, only for Workstation 6.0, as you mention yourself. Also, in terms of latency, the legacy API didn't provide better performance than the current VMCI Socket API, unless a combination of shared memory and polling was used.

2) The latency of the socket API is comparable to that of regular networking - for larger messages VMCI Sockets has a small advantage due to higher bandwidth, but for small messages the latency is about the same.

3) Currently, there isn't any other mechanism for VM to VM communication with lower latency than VMCI or networking. It would be interesting to hear a bit more about your use case; and what your expectations for low latency are, so that we can improve the APIs in the future. For asynchronous notifications between VM, there is a rather high cost regardless of the API, since one VM needs to first ask the hypervisor to interrupt the other VM, followed by the overhead of actually generating an interrupt in the receiving VM. These costs can be brought down by resorting to polling, but this usually results on higher CPU overhead.

Thanks,

Jorgen

AndrewSK
Contributor
Contributor

Jorgen, thanks for your answers!


1) The legacy VMCI API is already deprecated and was never released for ESX, only for Workstation 6.0, as you mention yourself. Also, in terms of latency, the legacy API didn't provide better performance than the current VMCI Socket API, unless a combination of shared memory and polling was used.

It's a pity Smiley Sad

Partially - yes, i'd a plenty of thoughts about a kind of a pooling or lock-free queues. O'coz, there is a lag of the OS thread slice in case of a large wait, but in cost of the cpu load (spinning, for example) we could reduce lagging.


2) The latency of the socket API is comparable to that of regular networking - for larger messages VMCI Sockets has a small advantage due to higher bandwidth, but for small messages the latency is about the same.

Smiley Sad


3) Currently, there isn't any other mechanism for VM to VM communication with lower latency than VMCI or networking. It would be interesting to hear a bit more about your use case; and what your expectations for low latency are, so that we can improve the APIs in the future. For asynchronous notifications between VM, there is a rather high cost regardless of the API, since one VM needs to first ask the hypervisor to interrupt the other VM, followed by the overhead of actually generating an interrupt in the receiving VM. These costs can be brought down by resorting to polling, but this usually results on higher CPU overhead.

Yes, i understand, that this context switch is not for free Smiley Happy But we expected (with multicore cpu) a possibility not to switch a context by using shared memory and spinlock-based sync. Ideally (if well designed), it could be compared in terms of performance with interprocess communication.

What's about our task. We have a number of lightweight and fast Smiley Happy producers of the notifications (each is placed in it's own guest) and we want them to be synchronously processed by single consumer on the other guest os. It's important that messages should be processed synchronously and as soon as possible - it could affect performance of whole guest. The amount of data... example of pattern - fully enabled procmon.

Maybe, there are any halfdocumented ability to establish shared memory between guests? We could discuss it privately, if there is any - in favor of vmsafe advancement Smiley Happy

Jorgen, thanks again for answers, it was really helpful!

0 Kudos