In this video interview, Jeff Buell, one of VMware's senior performance engineers, shares his perspectives on the importance of Big Data for customers and for VMware. He also addresses the issue of why his Hadoop testing showed significantly better performance running Hadoop on vSphere when compared to native in some instances. The video (running time 6:20) is here.

The full white paper detailing Jeff's Hadoop performance testing done in partnership with AMAX and Mellanox is here.

The video of the Big Data panel at the OpenFabrics Alliance User and Developer Workshop has now been posted at insideHPC.com -- it is available here.

DK Panda from Ohio State University spoke first and gave a compressed version of a much longer talk that illustrates the benefits of high bandwidth and low latency interconnect for distributed Enterprise services like memcached as well as benchmarks like Yahoo's Cloud Serving Benchmark (YCSB) -- interesting stuff, especially as an illustration of the converging requirements we are seeing between HPC and Enterprise.

My talk starts at about 12m30s in the video and, embarrassingly since it was supposed to be a 10-minute talk, ends at about 29m30s. I made some general comments about the larger context of Big Data, presented results of our performance tests running Hadoop virtualized on the vSphere platform and then ended with some comments about the role of interconnects for Big Data.

Milind Bhandaskar, EMC/Greenplum Chief Engineer, spoke next about how applications drive system development, specifically in the areas of machine learning, analytics and reporting as well as visualization for BigData. He also explained the phases seen in Data Sciences workloads: Obtain, Scrub, Explore, Model, Interpret.

The final panel speaker was Sumanta Chatterjee, VP Server Technologies at Oracle who spoke about Oracle's view of a BigData appliance built around an Acquire -> Organize -> Analyze workflow that uses Oracle Exa* products. He then spoke about specific issues with respect to BigData and bandwidth.

Bhavesh and I gave our talk [PDF] this morning at the OpenFabrics Alliance Developer and User Workshop here in Monterey. Actually, to be fair, I did a short introduction and then Bhavesh spent the bulk of our thirty-minute slot covering two topics. First, he presented several possible approaches to enabling RDMA for latency-sensitive applications on vSphere -- including what he called "the holy grail" -- a virtualized RDMA device that provides low-latency, high-bandwidth RDMA while maintaining the ability to use vMotion and Snapshots. In his view, this vRDMA option (Option F in the slide deck) is the most compelling and interesting for VMware and is the one he will be prototyping within the Office of th CTO over the coming months.

The second half of Bhavesh's talk focused on the potential benefits of using RDMA within VMware's virtualization platform to accelerate some of our important hypervisor services -- for example, vMotion and FT. The slide deck includes experimental results that demonstrate vMotion operations can be completed much more quickly and with far lower CPU utilization by using RDMA.

In my introduction, I showed a few graphs from our upcoming paper, RDMA Performance in Virtual Machines using QDR InfiniBand on VMware vSphere 5 that showed we can use passthrough mode (VM DirectPath I/O) to deliver 1.75us half ping-pong latencies for Send and 3us for RDMA Read using polling completions and 7.6us using interrpt completions for RDMA Read.

insideHPC taped our session so I will post a link once it is available.

In a recent blog post, I offered Tuning tips to run low-latency applications in virtual machines. The blog post and the Best Practices white paper highlighted that by following certain tuning tips, a broad array of I/O latency sensitive applications which were historically difficult to virtualize due to overheads and added latency can now be successfully deployed on VMware vSphere 5.0.

I'm excited to annouce that my colleage Shilpi Agarwal, who manages the vSphere Network Performance Engineering team, just published a companion white paper that examines Network I/O Latency on VMware vSphere 5.

The paper is an excellent and highly recommended read to learn about the exact sources of latency overhead in the vSphere network virtualization stack. It also includes specific ping and netperf (common networking benchmarks) numbers comparing latencies achievable on physical non-virtualized environments versus vSphere VMs on the same hardware.

The paper compares latencies between a VM on an ESXi host connected to a non-virtualized physical host, two VMs on the same ESXi host connected across a vSwitch, as well as two VMs on two ESXi hosts using DirectPath I/O. It also measures the impact of CPU and network resource contention on latency in vSphere 5.0.

This white paper, along with the previously published Best Practices for Performance Tuning white paper, and the soon to be published research note summarizing our QDR InfiniBand performance study using DirectPath I/O on vSphere 5, all collectively demonstrate how VMware vSphere 5.0 is finally a hypervisor platform that's viable for a significantly increased number of latency-sensitive applications in a virtualized environment.

I'm looking forward to attending the OpenFabrics Alliance annual workshop again this year. It's being held in Monterey, California March 25-28, 2012. Details are here.

Last year was the first time I attended the workshop. I talked about VMware's interest and requirements for using RDMA in a virtualized environment and I found the atmosphere very conducive to good conversations with fellow attendees. I found everyone very welcoming and interested in what VMware wants to accomplish. To me, this is the venue that represents the locus of activity around low-latency, low-overhead communication.

This year my colleague Bhavesh Davda (also from the Office of the CTO) will be attending as well. We will present performance results for our initial InfiniBand work and also discuss issues related to building a virtualized RDMA device, a project that Bhavesh is running within VMware. We've made significant progress since last year and I'm eager to share the results.

In addition to presenting what we've been doing around RDMA, I will also be participating on a Big Data panel session with the working title "Big Data – Shared or Unified, Local or Remote Access".