Munich / Mountain View
. The BMW Group Technology Office USA in Mountain
View presents a research update with Meta’s Reality Labs Research,
demonstrating, for the first time, the ability to accurately display
stable virtual reality (VR) and mixed reality (MR) content to
passengers in a fast-moving car, even when making turns, going over
speed bumps, and accelerating. By integrating Meta Quest’s tracking
system with BMW’s sensor data, the researchers enabled a series of
stable “car-locked” gaming, entertainment, productivity, and
meditation experiences.
Claus Dorrer, Head of BMW Group Technology Office USA in
Mountain View: “It is too early to tell exactly how or when
this technology will make it into customers’ hands, but we envision a
number of potential use cases for XR devices in vehicles — from
assisting the driver in locating their car in a crowded parking lot to
alerting them to hazards on the road and surfacing important
information about the vehicle’s condition. The implications of future
AR glasses and VR devices — for passengers as well as drivers — are
promising. The research partnership with Meta will allow us to
discover what immersive, in-vehicle XR experiences could look like in
the future and spearhead the seamless integration of such devices into cars.”
Proof-of-concept prototype overcomes key technical
challenges.
VR headsets use cameras and motion sensors
to understand their exact position, so when the user moves their head
or looks around, the virtual content remains stable. However, in
fast-moving vehicles, the headset’s sensors get confused by
conflicting signals from the device’s movement within the car and the
vehicle’s movement in the world. Simply put, the headset’s motion
sensors think it is moving, but its cameras think it is standing still
when looking at the car’s interior.
That mismatch means today’s VR headsets can’t display stable
virtual content when travelling inside a fast-moving vehicle,
particularly if the vehicle accelerates, turns fast, or encounters
bumps on the road.
To solve this challenge, BMW and Meta researchers collaborated
to incorporate information from a BMW car’s sensor array into the
tracking system of a Meta Quest Pro. That allowed the tracking system
to simultaneously calculate the device’s location relative to the car
and the moving car’s location relative to the world, enabling it to
accurately anchor virtual objects to a moving car and demo compelling
VR and MR passenger experiences.
Richard Newcombe, Vice President of Research Science, Reality
Labs Research: “Our research prototype shows that we can
enable entertaining and comfortable passenger experiences that are
anchored to the car itself, including VR and MR gaming, entertainment,
productivity, and even meditation capabilities. The technology has the
potential to transform how we can safely interact with our environment
while traveling, and as we progress into reliable world-locked content
on the road to AR glasses, we’re hopeful it will be possible for
passengers to see things like markers for landmarks, restaurants,
places of interest, and more.”
Combining the movement data generated by the vehicle with XR
headset sensor capabilities allowed us to subtract the vehicle motion
and more accurately track the headset’s movement in a vehicle. That in
turn enabled us to render content which is synced to the movement of
the vehicle in real-time, both inside and outside of the vehicle at
high speeds, regardless of whether the vehicle is on a flat, straight
street or a bumpy road with sharp turns.
This proof-of-concept prototype demonstrates an in-car full
range of a person’s movements (known as six degrees of freedom) and an
XR experience that simultaneously calculates a headset’s location
relative to the car and the moving car’s location relative to the
world by integrating with the car’s sensors.
The BMW Group has been successful in pioneering digital
technologies like vehicle access based on the Digital Key and in
driving industry standardization for such innovations. The company’s
global network of technology offices has been crucial in kickstarting
such innovation projects. The BMW Group is open to including
additional partners in order to achieve the goal of creating an
industry standard for connecting XR consumer electronics devices to
vehicles. For potential series development, the BMW Group is committed
to the privacy-by-design principle. This means, in all digital feature
development, the BMW Group takes steps to ensure that customer data is
protected and securely processed in accordance with data privacy
requirements by means of established processes — across all the
markets in which the company operates.
Link
to Meta Reality Labs information on the project.
