Smart Rendering for Virtual Reality

Virtual Reality for consumers is becoming more and more common. There are now several mobile solutions that transform your smartphone into a virtual reality device, by adding a case with wide-angle lenses attached to it. Also there are dedicated devices for your PC with additional positional tracking, so you can look around corners by moving your head. These devices deliver a highly immersive experience. However, single pixels from the screen can still be noticed. According to researcher Michael Abrash, the screen resolution needs to go up to at least 8Kx8K. That means 64 megapixels need to be rendered, compared to the 4 megapixels we have in today’s best consumer head-mounted displays (HMDs).

Getting the performance to render 16 times more pixels will not be straightforward. To achieve this, we better start looking into smarter ways of rendering. As of now, we treat the image rendered for a monitor almost the same as an image rendered for Virtual Reality. But inside an HMD there are different things happening, that we should not ignore. We talked previously about the radial distortions and chromatic aberration that are caused by the lenses of the HMD and how to handle those in a more advanced way without sacrificing sharpness. But when you look through an HMD there are other distortions as well, which have not been considered during the rendering process so far. Let’s see how the image you see looks like:

Astigmatism in HMDs

Due to the distortion astigmatism, only the center of the image can be perceived very sharp, while with increasing distance from it, it gets more and more blurred. Ideally we would like to minimize this effect, but this could only be done by adding more lenses, which is counter-productive towards the idea of having a cheap and lightweight HMD. Therefore, in practice, almost all new consumer wide-angle HMDs just use one lens.

As the center area can be seen best and the outer areas become blurred, why should we still spend the same performance and quality on each pixel?

We change this behavior by introducing the concept of sampling maps into our research ray tracing renderer (using Intel Embree internally).

Sampling Maps: Without and with Interpolation.

The sampling map describes for each pixel how much supersampling (anti-aliasing) should be maximally computed. White in the center means that the maximum (e.g. 4 primary rays per pixel) are desired. In the outer regions we continuously decrease to 1 ray per pixel. To get a smooth transition we interpolate the sampling map. Let’s see how this impacts the performance on our test system*.

For the following, we assume a maximum of 4x supersampling (4 primary rays per pixel). If we render the full image (without the black borders that come from image warping against radial distortions), we achieve 14 frames per second (fps). If we skip rendering the areas with black borders, we get 16 fps. Using sampling maps, so only in the center 4 rays per pixel are used and less rays in the outer areas, we double the performance to 32 fps.

Next, we did a user study to find out how much of the image users can see inside the HMD. More details will be in our full publication.

Maximal visible area

The image above marked the maximal visible radius that users were able to see in the Oculus Rift DK1. For lens A users (no vision correction of farsighted), there is some area that no one was able to see. For lens C (highly shortsighted) this is even more dramatic, as the field of view goes down when using that vision correction and more pixels are becoming irrelevant to the viewer. We analyzed what happens, if we take the maximal radius for lens C users, where 90% of the users would not see any more content. The remaining 10% could tweak the radius in a user-specific calibration step.

Optimizing for lens C users boosts the performance from 32 to 46 fps.

Another optimization that we worked on was a hybrid rendering system that uses lower quality rasterization (GPU-accelerated) in the outer areas and higher quality ray tracing (using a warped camera sampling pattern for maximal sharpness) in the inner, sharp area.

Hybrid Rendering: Using lower quality rasterization in the outer areas and higher quality ray tracing inside (red area)

Performance-wise, this technique runs at 54 fps in our test setup.

For completeness, we also look at the values if no supersampling is enabled, meaning we always shoot one ray per pixel (in the ray traced area). Doing so in the hybrid rendering approach leads to 116 fps.

Many of the approaches mentioned here are only viable if a custom-renderer exists, which can be modified to allow such an individual sampling of the scene. But some of these ideas can be directly adapted to today’s hardware-accelerated rendering pipelines using DirectX or OpenGL. For example, when warping the image in a post-processing shader to compensate for radial distortions, the inner area could use superior bicubic filtering, the outer area the standard bilinear filtering. Other fullscreen post-processing effects like Depth of Field (DoF) could use a better kernel in the inner area, and a faster version in the outer area, which is not perceived sharp.

More Details will be available at end of March 2015 in our presentation of “Using Astigmatism in Wide Angle HMDs to Improve Rendering”, by Daniel Pohl (Intel), Timo Bolkart (Saarland University), Stefan Nickels (Intel Visual Computing Institute), and Oliver Grau (Intel) at IEEE VR 2015.

Update: We were asked what happens if the user moves the eyes around inside the HMD. This does not influence our rendering algorithm as it is tweaked towards the lens astigmatism of the HMD. The eye position does not matter for this. The user study, that measured the maximal visible area, was done with instructions to the users to also move their eyes around.

* Test system: Rendered at 1280 x 800 (Oculus DK1) in a custom ray tracer on dual-socket 3.1 GHz Intel Xeon E5-2687W, NVIDIA GeForce 680 GTX