What is Spatial Computing?

Spatial computing uses cameras and sensors to create a digital model or digital twin of people, objects, machines, and the environments they’re in, to enable users to interact with them. Interaction and control in spatial computing often uses gestures, body movements and/or voice commands. Spatial computing combines, and builds upon, elements of virtual reality (VR), augmented reality (AR), mixed reality (MR) and digital twin technologies.

Spatial technology isn’t brand new, but it’s getting a renewed moment in the sun thanks to Apple’s latest device, the Apple Vision Pro headset. Apple’s launch messaging claims, “The era of spatial computing is here, where digital content blends seamlessly with your physical space. So you can do the things you love in ways never before possible.” 

Spatial computing is a broad concept addressing the ways people and technology interact, but in this specific case it’s also the latest interface technology being promoted as the future of Human Machine Interface (HMI).

From touchscreens to touch-free interaction

There was a time, not that long ago, when touchscreens quite suddenly revolutionized consumer HMI. To get a feel for how groundbreaking touchscreen technology felt early in its mainstream lifecycle, skip to the 33:10 mark of the iPhone 1 announcement to hear audible gasps from the crowd when Steve Jobs showed off pinch to zoom multi-touch functionality for the first time.

But the sea change brought on by the touchscreen has yet to give way to the next evolution of HMI tech. Spatial computing has a lot of potential in both the consumer and industrial space, but to understand where it might lead, we need to touch on how we got here.

Virtual, augmented and mixed reality vs spatial computing

First, let’s look at how the concepts of virtual, augmented, and mixed reality relate and differ. Fortunately, much of the necessary context to understanding the key differences is right there in the names.

Virtual Reality (VR) gives access to, or serves to place the user in, a completely digital, or virtual, space. Virtual reality can be:

• Non-immersive, like traditional video games played on a display.
• Semi-immersive, which places the users in a partial virtual environment, like a flight simulator with wrap-around displays and more true-to-life control devices.
• Fully immersive, most commonly using a headset that limits the user’s field of view to only the VR experience and folds in other sensory elements like spatial audio and haptic feedback.

Augmented Reality (AR) layers digital information and virtual elements on top of the user’s real-world environment to augment their perception and experience. A consumer example of AR would be Amazon’s “View in my room” feature and its “Room Decorator” counterpart, which uses your phone’s camera to place a virtual version of an item, or items, you’re interested in purchasing into the room or space where they’d eventually live. In the industrial space, manufacturers are using AR to train employees on operations and maintenance.

Mixed Reality (MR) builds on AR by enabling the intermingling, or mixing, of real and digital elements. Rather than just overlaying images or renderings, MR allows users to interact with the digital experience. One ubiquitous consumer example of MR are photo app filters, which can apply makeup or change your hairstyle and then respond to your movements and continue to apply the effects. Meanwhile, companies are employing MR technologies to assist with remote technical support.

Each of these technologies fall into the broader category of Extended Reality (XR), the umbrella term for this type of interplay between users and the digital space that enhance, augment or replace our view of the world. 

Spatial computing evolves from augmented reality

It had seemed, for quite a while, that augmented reality would be the next great leap forward in computer interface design. In fact, the concept of mixing visual graphics and information with the surrounding environment has been around for decades. Efforts to harness the potential of AR range from augmented reality mirrors for smart retail applications, to AR-assisted surgical procedures. However, some of the impediments to the growth of AR have included hardware and development costs, the often clunky headset peripherals packaged with AR solutions, and a less than ideal user interface experience.

Spatial computing addresses many of those user interface challenges by taking the impetus off of the user to learn new behaviors. By creating digital versions and 3D models of objects, people, and locations using various inputs including video, GPS, and LiDAR (among others), spatial computing creates a digital map of the defined space, which might be a room, a building or even an entire city. From there, software is able to orient the user in that space and react to their actions and movements to influence the digital representations they see, which can then, in turn, be used to influence the physical devices in the space.

By layering on 3D mapping and aspects of digital twin technologies, spatial computing provides more intuitive, natural interactions between the user and the digital world. This control methodology may only require the user to make a specific hand or eye movement, turn their head a certain way or gesture in one direction or the other.

For examples of some of the input gestures that can be used for spatial computing control, you can watch this video from the Apple Developer website about design for spatial input.

What is the future of spatial computing?

On the surface, the concept of spatial computing conjures images of Tom Cruise waving his hands at a virtualized control interface in Minority Report. In reality, spatial computing applications have far more potential in science fact than science fiction. It’s far too early to tell if we’re at the tipping point for the wider adoption of extended reality technologies, but the application potential is exciting.

By more directly intertwining digital information with users, physical devices and locations, there are clear opportunities for innovation in process optimization, cobotics, factory automation, and other industrial technologies. Is spatial computing something you’re interested in, or are you working on a spatial computing project that requires computing power? We’d love to hear more about what you’re working on, so don’t hesitate to reach out to our team.