mmWave MIMO Based Depth Maps for Wireless AR/VR

The mmWave communication system, deployed at the AR/VR device, is jointly leveraged for sensing and depth map construction.

Key ideas

  • Depth map construction sensing framework using mmWave transceivers
  • Under the constraints imposed by the communication hardware and frame structure
  • Sensing beam codebook specific for the depth map construction objective
  • Leveraging tools from successive interference cancellation
  • Joint beam processing approach for constructing high-resolution depth maps


  • Joint sensing and communication in AR/VR, smart home, or auto drive devices
  • Depth maps using mmWave MIMO sensing solution integrated with RGB-D cameras

More information about this research direction

Paper: Abdelrahman Taha, Qi Qu, Sam Alex, Ping Wang, William L. Abbott, and Ahmed Alkhateeb, “Millimeter Wave MIMO-Based Depth Maps for Wireless Virtual and Augmented Reality,” in IEEE Access, vol. 9, pp. 48341-48363, 2021.

Abstract: Augmented and virtual reality systems (AR/VR) are rapidly becoming key components of the wireless landscape. For immersive AR/VR experience, these devices should be able to construct accurate depth perception of the surrounding environment. Current AR/VR devices rely heavily on using RGB-D depth cameras to achieve this goal. The performance of these depth cameras, however, has clear limitations in several scenarios, such as the cases with shiny objects, dark surfaces, and abrupt color transition among other limitations. In this paper, we propose a novel solution for AR/VR depth map construction using mmWave MIMO communication transceivers. This is motivated by the deployment of advanced mmWave communication systems in future AR/VR devices for meeting the high data rate demands and by the interesting propagation characteristics of mmWave signals. Accounting for the constraints on these systems, we develop a comprehensive framework for constructing accurate and high-resolution depth maps using mmWave systems. In this framework, we developed new sensing beamforming codebook approaches that are specific for the depth map construction objective. Using these codebooks, and leveraging tools from successive interference cancellation, we develop a joint beam processing approach that can construct high-resolution depth maps using practical mmWave antenna arrays. Extensive simulation results highlight the potential of the proposed solution in building accurate depth maps. Further, these simulations show the promising gains of mmWave based depth perception compared to RGB-based approaches in several important use cases.

author={Taha, Abdelrahman and Qu, Qi and Alex, Sam and Wang, Ping and Abbott, William L. and Alkhateeb, Ahmed},
journal={IEEE Access},
title={Millimeter Wave MIMO-Based Depth Maps for Wireless Virtual and Augmented Reality},

Having questions or feedback?