Virtualization performance testing tips

posted by John Troyer (VMWare) on August 20, 2008

I attended an interesting Chalk Talk today from Intel's Kshitij Doshi and Ashok Emani: "Taming the Complexity of Studying Performance Under Virtualization". They walked through the development of a server consolidation study usng the vConsolidate Framework. Performance testing of virtualized systems is fraught with complexities, especially when you're trying to say something about real-world scenarios with multiple virtual machines.

Kshitij and Ashok walked us through their methodology and experiences. To oversimplify, when you're dealing with multiple workloads on top of a hypervisor, you not only have many points of failure (like apps or testing frameworks crashing), you also now have many ways for bottlenecks to arise (in CPU, disk, I/O, memory), and often the hypervisor helpfully hides those bottlenecks in an attempt to do production-level resource allocation over all its virtual workloads.

Their list of "common sources of errors and anomolies" is worth a paper of its own, as you can tell it comes from long experience, but for this blog post let me just hit the headers of their slides on "common pitfalls." After reading this, I hope you will think twice before just firing up a quick timer on a process in a virtual machine. It's probably not telling you what you think it's telling you! (Most real-world virtualized workloads are not performance-bound, anyway, but that's a whole other conversation.)

• Time drift
• Unmonitored failures (loadsim, webbench)
• Disk space
• Spurious interrupts, network isolation
• IOPS contention
• Client instabilities (memory leaks, MTU, login failurers)
• VMM "knobs"
• Guest OS knobs -- Ticklessness (avoiding context switches)
• Application tuneables -- Java heap and large page tuneables
• Affinitization (shared caches on multicore machines can help or hurt)
• Service packs and application versions
• Client memory exhaustion
• VMM memory fragmentation, oversubscription
• BIOS or hardware "knobs"

For more information on vConsolidate, check out this paper from the Intel Developer Journal, Redefining Server Performance Characterization for Virtualization Benchmarking. Tom Adelmeyer gives some context in this video presentation on the Intel Software Network: Virtualization Performance Testing with vConsolidate.

Over at VMware, we have also developed a framework known as VMmark, and our performance team often publishes results at VROOM!, the VMware performance blog, which is interesting reading if you're a performance geek or just trying to get a sense of how hypervisors scale in 2008. Both VMware and Intel are also working together with SPEC to develop an industry-wide benchmark for virtualized systems.

Comments (3)

tagged: benchmarking, idf, idf2008, performance, virtualization