High-Performance Virtualized Spark Clusters on Kubernetes for Deep Learning

High-Performance Virtualized Spark Clusters on Kubernetes for Deep Learning

A virtualized cluster was set up with both Spark Standalone worker nodes and Kubernetes worker nodes running on vSphere virtual machines. The same Spark image classification workload was run on both Spark Standalone and Spark on Kubernetes with very small (~1%) performance differences, demonstrating that Spark users can achieve all the benefits of Kubernetes without sacrificing performance.

See the paper High-Performance Virtualized Spark Clusters on Kubernetes for Deep Learning​ by Dave Jaffe.

Comments

thanks for sharing man!

Version history
Revision #:
1 of 1
Last update:
‎11-19-2019 04:45 PM
Updated by: