Nicole Hemsoth, who started HPC in the Cloud and who is now editor of Datanami which focuses on Big Data, suggested I visit the Cisco booth and find out what Jeff Squyres has been working on.
I've known Jeff since the days when I evaluated Open MPI and proposed successfully that Sun drop its proprietary MPI implementation and join Jeff and others on this then-emerging open source implementation.
At SC11, Jeff showed he could achieve half-roundtrip latencies on ethernet of about 5us, user process to user process. He did this on a native system (no virtualization) using a polling, user-mode driver written to talk directy to the Cisco 10 GbE Palo adapter (officially known as the Cisco UCS M81KR Virtual Interface Card) on UCS systems. The latencies were great, but we talked about something even more interesting.
The Cisco Palo cards were co-designed with VMware's vmxnet3 virtual device to share a common data path specification. Which means that in addition to demonstrating excellent performance, we should be able to migrate running VMs that are using Palo adaptors in passthrough mode (VMware Direct Path I/O) on vSphere 5. For information on how that works, see this presentation [PDF].
Using Jeff's user-mode driver running in a virtual machine on vSphere 5, we should be able to demonstrate low latency, high-performance Ethernet-based MPI application results. And, in addition, we should be able to start experimenting with providing proactive fault resiliency with Palo and UCS for single and multiprocess applications -- that is, the ability to proactively move workload or pieces of workload to avoid underlying hardware failures.
