Facebook has been awarded a patent for a head mounted display (HMD) which combines a large low resolution display and small high resolution display projected to where the user’s eye is pointed to achieve ‘retinal’ resolution.
‘Retinal’ or “retina” is a term often used to describe angular resolution which at least matches that of the center of the human eye.
Facebook is the company behind the Oculus brand of VR headsets and services. Originally purchased as a startup in 2014, Oculus is now a division of Facebook. This patent’s inventors are all listed as residents of Washington state, suggesting this idea comes from Facebook Reality Lab which has its main office there.
Two Displays Per Eye, Merged
The patent describes a headset which has eye tracking-driven foveated rendering. For those unfamiliar, foveated rendering is a process which renders most of the view into a virtual world at lower resolution except for the exact area directly in front of where the user’s eye is pointed. That area in front of the eye — where humans perceive the greatest detail — is rendered at a higher resolution.
With this patent, instead of the image being sent to one display per eye, as in most headsets, the high resolution area is instead sent to a a second much smaller display called the ‘inset display’. A steerable mirror and optical combiner then project this display into the lens, at the position the user’s eye is pointed. Low resolution parts of the virtual world — parts not directly in front of the eyeball — go to the main display and are magnified directly by the lens.
The result would a display that combines these low and high-resolution panels to provide an experience that roughly matches the level of detail that the human eye can resolve. If the eye tracking is good enough, the user would not even notice that the headset has variable resolution.
Isn’t This Varjo?
This patent may sound familiar if you’ve heard of the Finland-based company Varjo. Varjo’s current prototype also features an inset and background display, but the high resolution area is locked to the center of the display — it does not yet adapt to eye position. But Varjo’s end goal is to build a headset that sounds surprisingly similar to what Facebook describe in this patent, steering the display with mirrors.
Varjo has also been awarded a patent for this technique. Facebook applied for its patent before Varjo’s, but Varjo’s was granted before Facebook’s. It is not clear how much these techniques differ from one another.
The ‘Inset’ Microdisplay
One diagram in the patent’s supporting documents mentions the resolution and potential supplier of the inset display. It is marked as a 1920×1200 microdisplay from eMagin. This is likely the eMagin WUXGA, which the company claims is the highest resolution OLED microdisplay in production.
OLED microdisplays use a more costly production method compared to regular OLED panels used in VR today, but are physically much smaller and consume less power. The peripheral display’s exact resolution is not listed, but is described “low compared to other displays”.
Staggering Angular Resolution
In another diagram is marked the vertical field of view of the projection of the inlet display – 17 degrees. Given that we know its vertical resolution is 1200, that could mean that it would provide an average vertical angular resolution of roughly 70 pixels per degree (PPD). Oculus Go, the company’s current highest resolution headset, has an angular resolution of roughly 15 PPD.
Achieving this kind of PPD is not yet possible with traditional VR display systems, as there are no regular OLED displays with even close to the resolution that would be required. This resolution requirement would get progressively higher for higher field of view optics, to the point of impracticality. Even the hypothetical 4000×4000 per eye headset with 140 degrees field of view predicted in 2016 to exist by 2021 by Oculus Chief Scientist Michael Abrash would have only around 30 PPD. The current highest resolution VR-suitable OLED on the market is Samsung’s 1440×1600 panel, used in the HTC Vive Pro and Samsung Odyssey series.
Promising, But Just A Patent
It’s important to note that companies patent techniques all time and most never make it to a consumer product. Color microdisplays of this resolution can cost thousands of dollars, and it’s unclear whether mass production would bring them down to an acceptable cost. There may also be other challenges in manufacturing the complex optical combination system this patent describes.
