RAS and virtual infrastructure: keeping all your virtual eggs safe

posted by John Troyer (VMWare) on August 19, 2008

Hi, I’m John Troyer, covering IDF from the VMware side. I normally blog over at VMTN Blog, but this week I’m at IDF and blogging here to look at what Intel and VMware are doing to push forward the cutting edge of virtualization.

I managed to catch a good chalk talk today, VIRC002, “VMware Virtualization Taking Advantage of Server Reliability, Availability, and Serviceability Technologies.” Here’s an overview of RAS from Wikipedia if you’re not familiar with these concepts that came out of the mainframe world. (And let me give a disclaimer that all errors in my summary here are mine only, as I’m usually more of a software guy and memory errors for me usually revolve around where I left my keys.)

Rich Brunner from VMware touched on a number of topics, but the basic challenge he talked about is the “all your eggs in one basket” problem as we build the datacenter of the future. As CPUs become more capable, virtual machine density becomes higher. Imagine a future 8-way server, with each processor having 16 cores (I said future), and 8 virtual machines on each core. That’s 8 x 16 x 8 = 1024 virtual machines on this hypothetical future piece of hardware. In this future, one memory error can crash the whole physical server, bringing down 1024 virtual machines. Yes, you will have your second hot server standing by, but it seems like we should be able to do better than crashing all 1024 virtual machines for an itty bitty memory error.

It turns out Intel, VMware, and the hardware OEMs are working on a number of ways to be smarter about small point failures on large systems. The hypervisor will be able to do more predictive RAS as the components of the system start to cooperate — flag this correctable error; note that this other memory node is starting to fail, so stop using it for VMs; start evacuating it to other physical memory, and alert the management interface. The hypervisor can present a view of the hardware to VMs that only includes the parts that are working correctly.

Rich also talked about future improvements in data poisoning, where the hypervisor could track which VMs have been ‘poisoned’ by the bad data and restart only those, or if the bad data was pulled into the CPU predictively, perhaps no VMs have used it, and so they can run merrily along without disturbance and the error taken care through other means.

Finally Rich talked about Record Replay, a technology that you should expect to see more of at VMworld 2008. This is a technology that records every CPU instruction from a virtual machine which can then be “replayed” on another virtual machine, CPU instruction by CPU instruction. You can see this technology today as part of the VMware Workstation 6.5 beta, and you can even attach these replay sessions to a debugger to shine a light on your hard-to-reproduce “heisenbugs.” But what Rich was talking about was another use case that we demoed last year at VMworld, where the replay can be going on simultaneously on another physical server that can take over at that precise machine instruction if the first one has a problem. This isn’t a solution to all clustering and fault tolerance problems, but it does have the advantage of being at the virtual machine level, so it doesn’t matter what OS or application you’re running in your VM.

It was a good session looking at the future of RAS. It sounds like a vision that the industry will have to rally around — the makers of chips, servers, and hypervisors will need to agree on common interfaces and open standards to let our virtual machines live out “in the cloud” without worrying that a cosmic ray is going to crack all your eggs at once.

Comments (1)

tagged: idf, idf2008, RAS, virtualization