Unity Project for Hololens

Introduction

This repository contains the Unity project that is build and deployed onto the Microsoft Hololens.

Goals

• Establish ROS# Communication to a rosbridge server running on Linux box.
• Access camera stream in Unity and send images over ROS#
• Receive communication from ROS node with co-ordinates of faces and draw face pose estimations in AR
• Access Research mode sensor data in Unity

Media

Single Person Direction Indicator

Building The Project

• Uses this commit of dwhit's repository. Make sure to checkout this version.

Checklist

• Check Hololens IP
• Check ROS Computer IP
• Ensure ROS# Connector is connecting to the ROS Computer IP and using UWP mode
• Ensure MRTK build settings have been applied

Notes

• There is a significant improvement in responsiveness of the video stream when deploying to the Hololens in Debug or Release mode. Use Release mode for speed!

Viewing the Camera Stream

Accessing the camera stream is simple. Open rviz on the ROS Computer and subscribe to the image_transport/compressed topic.

Current issue with the video stream is that the image is mirrored horizontally.

Issues

Text Mesh Pro & Urdf

I opened an issue on the ROS# Project. The following was what I had to do to fix it.

For anyone else facing similar problems, I believe I have partly solved the issue. When I attempt to build the project in Unity, I get the two errors from above followed by a build complete:

Build completed with a result of 'Succeeded' UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

A solution pops up in the App folder, which I can open in VS2017. When I attempt to deploy the program onto the Hololens, the following happens:

This seems to be an issue with Textmeshpro.

This Stack Exchange thread solves the issue, and allows me to build and deploy the project to the Hololens. I can then check my rosbridge on the other computer and see a connection. This sort of explains why the files dont get copied over.