The Significance of Spatial Awareness for Spatial Computing in Augmented Reality
After the recent launch of Apple’s AR headset, Vision Pro, the terms “augmented reality (AR) and spatial computing” have once again emerged as transformative technologies. While it is still early days for AR, the launch of Apple’s headset has provided the necessary traction, and people have recognized its potential in seamlessly merging the physical and digital worlds to enhance user experiences. One critical aspect of AR is spatial computing, which entails the precise understanding and interaction of devices with the physical environment. Achieving spatial awareness, through the coordination of AR devices and the seamless sharing of spatial information is crucial for the success of spatial computing. In this article, I will delve into my work with Popvue on spatial awareness, done during the launch of ARkit in 2018, and explore the importance of spatial awareness, the challenges it presents.
Understanding Spatial Awareness
Spatial awareness in AR entails creating a unified coordinate system in which all devices operate and share spatial information with minimal latency. It involves accurately perceiving the physical world, tracking objects and users, and understanding their positions and movements relative to the environment. By achieving spatial awareness, AR devices can precisely overlay digital information onto the real world, creating a seamless and immersive user experience.
Spatial awareness allows multiple AR devices to overlay digital information on tracked objects in real-time. This technology holds immense potential for numerous use cases that will significantly impact our daily lives once AR and spatial computing become mainstream.
I was involved in a spatial awareness project called “Popvue,” back in 2018, which focuses on a specific application use case: efficiently connecting people during events. The primary function of this application is to track the real-time Pose of devices and seamlessly share this information with all other devices in the same map network. Additionally, it receives Pose data from all other devices within the network. Later in this article, we will delve into further details about the application’s functionality and how it operates.
Importance of Spatial Awareness
- Enhanced User Experience: Spatial awareness allows for a more immersive and intuitive user experience. AR applications can precisely align virtual objects with the physical world, creating the illusion of digital content seamlessly integrated into the user’s surroundings. This level of accuracy enhances user engagement and immersion.
- Collaborative Experiences: Multiple users can share a synchronized AR experience with spatially aware devices. For example, in collaborative design or gaming scenarios, users can interact with virtual objects in a shared physical space, facilitating seamless collaboration and enhancing communication among team members.
- Object Recognition and Tracking: Spatial awareness enables AR devices to identify and track real-world objects accurately. This capability opens up various use cases, such as industrial maintenance, where AR glasses can recognize complex machinery parts and overlay relevant information for troubleshooting or repair.
Project Popvue
Popvue was an event application made to connect people efficiently at an event site. At its core, it was a spatial networking platform on Edge, that enabled spatial awareness between devices while sharing an AR session.
The challenge of achieving spatial awareness in AR can be broken down into four steps:
Establishing a common coordinate system: When an AR application initiates a session, it creates its own arbitrary coordinate system. However, when multiple AR devices are present in a given space, each device operates with its independent coordinate system, lacking any correlation with the others. Therefore, the initial step in achieving spatial awareness was to synchronize all devices into the same coordinate system after initialization. This allows for the tracking of Pose through networking data shared by the devices within the area.
Real-time sharing of spatial information: Sharing spatial information poses challenges, primarily due to the substantial and complex data that needs to be transmitted and received per frame. Ensuring low latency in information sharing is crucial, as both physical and virtual overlays need to accurately track the movements of devices within the area.
Privacy considerations: Spatial awareness in AR raises concerns about privacy. If adopted on a large scale, there will always be cameras and sensors directed toward each user within the network. It is essential to ensure that camera and sensor data are solely utilized for tracking purposes, with no personal information leaving the device and no storage or tracking of data directly associated with personally identifiable information (PII).
Accessibility: The use case of spatial networking on smartphones may not be ideal, as users would need to activate their cameras to participate in the network, which also limits the possibility of background tracking. However, with AR headsets, this issue can be mitigated. These headsets offer a new form factor where cameras and other sensors, such as LiDARs, constantly track the 6DoF (Six Degrees of Freedom) pose of the device. This allows the application to seamlessly integrate into the spatial network at all times, eliminating the need for user intervention.
Challenges in Achieving Spatial Awareness
- Device Heterogeneity: AR devices come in various forms, including smartphones, headsets, and glasses, each with tracking mechanisms and sensors. Coordinating these devices within a unified spatial awareness system presents a challenge due to the differences in hardware and data processing capabilities.
- Latency: Spatial computing heavily relies on real-time tracking and rendering of virtual content. Any latency in sharing spatial information among devices can lead to misalignments between the virtual and physical worlds, resulting in a degraded user experience. Minimizing latency is a critical challenge for achieving seamless spatial awareness.
- Environmental Variability: The physical environment in which AR devices operate can vary significantly, presenting challenges for accurate spatial awareness. Factors such as varying lighting conditions, occlusions, and dynamic objects in the environment can impact the ability of AR devices to perceive and track objects accurately.
Conclusion
Spatial awareness is a vital component of spatial computing in augmented reality. It enables devices to accurately perceive and interact with other devices and users in the same map network, creating seamless and immersive user experiences for collaborative applications. Overcoming challenges such as device heterogeneity, latency, and environmental variability is essential for achieving full spatial awareness. The potential use cases of spatial awareness span various industries, including events, social networking, and payments. As technology advances, further progress in spatial awareness will unlock new possibilities, revolutionizing how we interact with each other digitally in the physical world.
