Wireless AR/VR with Edge/Cloud Computing

Overview

Triggered by several head-mounted display (HMD) devices that have come to the market recently, such as Oculus Rift, HTC Vive, and Samsung Gear VR, significant interest has developed in virtual reality (VR) systems, experiences and applications. However, the current HMD devices are still very heavy and large, negatively affecting user experience. Moreover, current VR approaches perform rendering locally either on a mobile device tethered to an HMD, or on a computer/console tethered to the HMD.

In this project, we discuss how to enable a truly portable and mobile VR experience, with light weight VR glasses wirelessly connecting with edge/cloud computing devices that perform the rendering remotely. We investigate the challenges associated with enabling the new wireless VR approach with edge/cloud computing with different application scenarios that we implement. Specifically, we analyze the challenging bitrate and latency requirements to enable wireless VR, and investigate several possible solutions.

Three possible approaches to realize the wireless VR/AR with cloud/edge-based implementation: (a) rendering on the cloud server; (b) rendering on the remote edge server; (c) rendering on the local edge device.

xueshi_p1.png

 

Solution 1: Multi-user Difference Encoding & Hybrid-casting to Reduce Bandwidth 

We propose a hybrid-casting approach to significantly reduce the bitrate needed to transmit the multiple user views, without compromising view quality. Specifically, we define the view of the user which shares the most common pixels with other users as the primary view, and the other views as secondary views. For each secondary view, we can calculate its residual view as the difference with the primary view, by extracting the common view from the secondary view. We term the above process of extracting common view and determining residual views multi-user encoding (MUE). Instead of unicasting the rendered video of each user, we can multicast/broadcast the primary view from the edge node to all the participating users and unicast each residual view to the corresponding secondary user. 

xueshi_p2_new.png

 

Solution 2: Achieving Ultra-Low Latency by Rendering/Streaming 360-degree Video 

 

As opposed to the VR approach where the field of view (FOV) is rendered, encoded and streamed only after new head rotation or body movement of the user is tracked and transmitted to the cloud/edge, in the case of the proposed 360-degree video approach, the 360-degree video is rendered in a regular interval, or with any changes in the user control information or virtual space change (not in response to user head movement), and streamed and cached at the user device. When the user subsequently performs head rotation, the new head position is tracked and used to select the appropriate video from the cached 360-degree video in the user device, and displayed on the user VR glasses, thus eliminating the delays associated with FOV rendering and streaming. The head rotation latency will be reduced to head tracking delay (less than 4ms with current technologies) and HMD display delay (less than 11ms with current displays), and hence meeting the ultra-low latency requirement of 20ms–30ms.

 

While this approach can significantly reduce head rotation latency, it can also significantly increase computation at the edge/cloud server (for rendering and stitching multiple camera views) as well as the bit rate needed to transmit 360-degree video for each user associated with the application session, which is about 4 times more bitrate than for 90-degree field of view). Therefore, we propose to investigate the use of multiuser encoding technique (MUE) to significantly reduce the bitrate needed to transmit the 360-degree videos to the users, as well as explore its use to reduce the rendering cost by rendering just the primary view and residual views of secondary users, instead of 360-degree views of all users.

 

xueshi_p3.jpeg

360-degree Panorama of virtual classroom

 

Associated Publications