Optimizing virtual reality experiences: Rendering in the cloud over 5G 

Client overview

Our Client is a maker of popular Virtual Reality headsets and has an eco-system of VR applications available directly from their app store.  For their mobile ready VR headsets, they also host a companion app on android and iOS platforms to communicate directly with the headsets.    

Our Client’s headsets work autonomously, without a wired connection. It’s a multifunctional device for entertainment and education that immerses its user in a virtual environment.  When you put on the headset, the functionality allows you to look around the outside world to understand exactly where you are before immersing yourself in the virtual world. You can play games or browse multimedia even on the go. In addition, you can participate in interactive training in a virtual environment. 

Optimizing virtual reality experiences: Testing next generation features 

Our Client began to explore ways in which they could leverage virtual machines in the cloud to handle all rendering remotely.  This sub-feature reached a level of maturity which prompted our Client to evaluate how viable this technology is in tandem with major 5G cellular networks (compared to home Wi-Fi networks).  Their focus was on major carriers like AT&T and Verizon, starting with a pilot program on Verizon for 3 months 

To evaluate this technology, our Client engaged us to hire testers to evaluate the new feature they had developed while connected to a 5G cellular network (via mobile hotspot).   

A team of 50 testers was recruited to play video games in virtual reality (VR) at specific time bands each day, with 25 to 50 testers playing VR games at specified times for up to 6 hours per day per tester. The testing was performed in various locations across California, USA based on Verizon service maps and our Client’s guidance. 

The key deliverables that were agreed with our Client included: 
    

• The recruitment and management of a team of up to 50 active testers who were able to deliver up to 200 hours of data per day. 

• The services were to be provided from home locaitons in the San Francisco Bay Area and managed by our teams. 

• The appointment of a team to oversee and manage this initiative. 

• Handling device shipping and retrieval logistics, and the monitoring of ongoing scheduling. 

• A buffer of up to 25 participants were recruited in addition to the number of participants needed to allow for no-shows, drop-outs and performance. 

• To track and report on participant metrics during the project. 

Navigating virtual worlds with comprehensive VR testing approaches

Highlights of our project execution approach included the following: 

• Tests were executed according to specifications. 

• Our Client had access to a reliable forum for technical questions and concerns. 

• The identification of feature gating/access requirements at beginning of the project to ensure smooth kick-off. 

• Access to feature-specific telemetry at the beginning of the project was given to enable vendor-conducted audits of participant activity. 

• The establishment of service level agreements for items such as communications to ensure cross-team alignment. 

• Alignment on reporting data to ensure process refinement. 

• The establishment of defect metadata to proactively monitor bugs/test-blocking issues in tasks. 

Key deliverables 

• 10,800+ testing hours were logged with 29,500+ submission forms submitted. 

• 57 global test reports, 17 targeted test reports, 2100+ unreliability reports and 6 activity trend reports were submitted. 

• 19 defects and bugs were opened and submitted 

• 50 participants within client-provided geo-locations were recruited. 

• Collaborated with engineering to establish process regarding user account creation, access, and management. 

• Successfully identified and mitigated a test blocker involving an OTA update channel, allowing participants to remotely update OS builds. 

• Cross-referenced telemetry, hours logged, and session report data to mitigate test blockers and identify gaps in coverage. 

• Re-hired and equipped 10 participants for the project extension in less than one week.