A detailed report from The Information charts the decisions behind the apparent years-long delay of Apple’s upcoming AR/VR headset.
In 2021 Bloomberg, The Information, and supply chain analyst Ming-Chi Kuo released reports claiming Apple is preparing to release a premium headset for VR and AR with high resolution color passthrough.
Apple’s VR/AR team is called Technology Development Group (TDG) and is led by former Dolby executive Mike Rockwell. As previously reported by Bloomberg in 2020, TDG originally developed a powerful “hub” which would wirelessly stream high fidelity content to a lightweight headset. But Apple’s former Chief Design Officer Jony Ive apparently “balked” at the idea, instead insisting the headset be standalone. The standoff between Rockwell and Ive reportedly lasted months, when CEO Tim Cook sided with Ive in 2019.
The decision to ship a standalone headset apparently left some members of TDG in “disbelief” because they had built high fidelity software designed to run on a powerful external processing unit, including photorealistic avatars.
The report details the challenges involved in trying to ship an ultra high end headset with all the processing onboard without inducing overheating or compromising battery life. Apple leadership apparently expect an AR experience “far beyond” what Mark Zuckerberg’s Meta can offer, and Rockwell is apparently struggling to deliver on this.
The headset apparently has 14 cameras in total when including those used for passthrough, positional tracking, eye tracking, face tracking, hand tracking, and body tracking – causing “headaches” for hardware and algorithm engineers. To handle this Apple built a dedicated image signal processor (ISP) chip called Bora, but engineers are apparently struggling to eliminate latency between it and the headset’s main processor – the upcoming M2. To mitigate this Apple apparently had to include a third chip to act as a “fast conduit”, but this hasn’t fully fixed the issue.
A unique apparent feature of the headset is a screen on the front to let others in the room see your upper face and eyes. But adding this screen apparently meant the passthrough cameras had to be placed in “awkward” positions far away from where the user’s eyes would be, making the development of passthrough reprojection algorithms more difficult.
The Information cites “several people familiar with the headset” as describing these makeshift solutions as a reflection of “overengineering, too-complicated solutions that often result from poor planning”.
While Jony Ive left Apple in 2019, the report claims he’s still involved with the project as a consultant, with employees needing to travel to his San Francisco home to get his approval on changes. Earlier prototypes apparently integrated the battery into the headband, but Ive “prefers” a design with a tethered battery the user wears. The Information says it doesn’t know which approach will be used in the final design.
The Information says “four people who have worked on the project” criticized the lack of focus on gaming. TDG apparently “almost never” mentions games in internal presentations, and isn’t developing a tracked controller, with the input methods instead being hand tracking and a “clothespin-like finger clip” which showed up in Apple patent filings.
The Information’s report earlier this week cited five sources revealing Tim Cook rarely visits the team working on the headset – a stark difference to Meta where some employees are reportedly frustrated at Zuckerberg’s obsession with VR and AR. The lack of a top level Apple executive championing the project has apparently made it harder to get engineering staff & resources allocated compared to the iPhone and Mac. To get support for the project, team members apparently warned that companies like Facebook and Magic Leap could end up owning the sector.
The Information previously reported Apple is considering pricing the product as high as $3000, but says top Apple executive Dan Riccio was assigned to the project last year to focus on reducing the cost of materials to make it more affordable. It should end up competing with Meta’s Project Cambria, slated to launch later this year for “significantly” above $800. If the reports so far are to be believed, though, Apple’s product will have higher resolution, a more powerful processor, and a slimmer design – though clearly achieving all that isn’t without challenges.
