6DoF tracking support #210
Conversation
Changed from CLOCK_MONOTONIC to CLOCK_BOOTTIME to be in line with the sensor timestamps and do pose prediction correctly.
|
I made this repository to show how to use this SDK in Unity: https://github.com/Aryzon/cardboard-xr-plugin/ see the instructions over there. |
sdk/sixdof/position_data.cc
Outdated
| @@ -0,0 +1,82 @@ | |||
| /* | |||
| * Copyright 2019 Google LLC | |||
sixdof-unity/SixDoFPoseDriver.cs
Outdated
| #if UNITY_EDITOR | ||
| public Transform editorPoseProviderTransform; | ||
| #endif | ||
| #if ARYZON_ARFOUNDATION |
There was a problem hiding this comment.
May you remove that specified flag?
| // Returns the latest value stored in the internal buffer. | ||
| Vector4 GetLatestData() const; | ||
|
|
||
| Vector4 GetInterpolatedForTimeStamp(const int64_t timestamp_ns) const; |
There was a problem hiding this comment.
May you add some description for this function as well?
|
Thanks for you comments! I implemented the requested changes and made a couple more improvements. BTW: This branch makes use of my 'multiple orientation support' implementation for which I made another PR. However the 6dof branch does not yet implement the changes suggested here: #208. What do you think of the way this is implemented in the head tracker? Do you think that is the right way to go? |
We have done an internal document to understand the implications of adding this extra feature to the SDK. I can comment and share specific parts of it so I'll limit myself to that. Overall, the design decision is to get the data from an external source and use it to provide it in the API. Given that your proposal here is aligned, would you like to engage with us on an implementation plan @maartenslaa ? |
|
Hello! I came across this as I was developing the same thing and was looking into whether PRs are typically accepted. Are you still developing this? I have a somewhat different approach for translation prediction that I was curious if you may have tried already. |
|
@feilen we are using our own implementation at the moment which we updated 7 months ago to be in sync with 1.19 of the Cardboard SDK: https://github.com/Aryzon/cardboard. Have a look at this if you want to use it with Unity3D: https://github.com/Aryzon/unity-sdk. Note we are using UI specific for our own AR headset, however you can scan any Cardboard QR code. I'm curious about your translation prediction approach, we have not yet tried anything else. |
|
To be honest, I have not dug into the ekf part enough to say whether or not that will work or gives different results. What I believe is the case with sensor fusion is that you feed it a new sample and it predicts from the current state and this new sample. Changing past samples may be a bit tricky and may require updating the entire state. Another thing to be aware of, ARCore and ARKit data is in essence sensor-fused data. You are inputting sensor-fused data into a sensor-fuser. I'm not sure how that will work and how useful that will be. Adding this 6DoF data as a new sensor to the sensor fuser may possibly be a better way but like I said, I do not know enough about sensor-fusion. In the end the results may be very similar to our current implementation. Have a look at how kReduceBiasRate is used in our implementation. Setting this to 1 should be the same as closing the drift gap using the latest rotation samples as difference, which means no drift. Our current implementation is not ideal since no neck model is applied to the rotation values, rather the 6DoF position data is extrapolated to the current time. The slower the 6DoF updates the worse this result is, even for simple head rotation. One thing I am thinking of right now is you can estimate a better neck-model with head rotations as long as the user sits still and only rotates their head / neck. It might be difficult to use a neck-model together with 6DoF position data however, I have not thought this through but it might require trying to separate body movement from head movement. With the device-pose offset, do you mean neck model? Or the difference in 6DoF position data and center screen position? Device-pose offset should be dealt with by the cardboard sdk right? Or am I missing something? Note: 6DoF position data is the difference in translation from the camera start position. Camera position is different for every phone. Sadly there is no API to get the center screen position for instance as a common reference point for every phone. This may however only be a problem for us since we are doing AR and need to project scanned data like surfaces or point clouds onto the real world at the correct position. It may not be an issue for VR. |
The 'sensor fuser' is just Kalman filtering, which lets you predict and update over noisy data. My addition of re-building the current position every time is just an unusual step to get around high latency, but should give you near-realtime responsiveness (since the major disadvantage to accelerometer based positional tracking is drift, for which only being ~150ms away from a ground truth should fix). I'm implementing it right now, I'll tell you how it goes!
Ahh, I only intended to use the neck model as a fallback if vision-based tracking is lost. With the fusion above, we wouldn't need to worry about responsiveness.
I specifically meant screen-surface-to-eyes offset, since the phone is generally in a holder. From the looks of it, your headset would require something a little more custom, but my approach was to simply subtract the lens-to-screen offset in the viewer parameters.
Frame.getAndroidSensorPose() in ARCore gives you the Android Sensor Pose, which is the center of the surface of the screen. The offset I'm describing is from there to your eyeballs, which is where the eye (camera) position pose should end up. |
|
I did not know about Frame.getAndroidSensorPose() in ARCore, very good to know this exists, thank you! I hope all Android phone manufacturers give the correct pose.. |
Hey!
I have been working on 6DoF support and got it working with Unity and ARFoundation. It should work with native ARCore and ARKit as well. 3DoF still works. See this commit for the changes 6DoF related.
The code adds functionality to the head tracker, you simply call 'AddSixDoFData(timestamp, postion, rotation)' and the head tracker merges this data with the data from the sensor fusion in GetPose(..).
For this to work you will need to call the AddSixDoFData(..) every time ARCore / ARKit updates. I have not tested this native yet, but have tested this in Unity with ARFoundation. Native should be a bit easier since no XRInputSubSystem loading is required.
Edit: To get this to work in Unity you will need to follow the instructions here: https://github.com/Aryzon/cardboard-xr-plugin/
The code is meant to be non-intrusive to 3DoF tracking and adds little overhead. It is flexible, you can add your own 6DoF implementation, simply call AddSixDoFData() every time you have new data. I did not implement ARCore and ARKit code into the SDK because I needed to support ARFoundation. It will otherwise be much more difficult to support the many subsystems that ARKit and ARCore provide to ARFoundation like plane detection, body segmentation etc..
One thing to note, I set Rotation differenceToSixDoF in the function AddSixDoFData(..), this Rotation is then used by GetPose(..). I can see this can be an issue when using different threads and I'm not sure I handled it correctly to support different threads.
About me: I'm one of the devs at Aryzon, we make mobile based MR headsets that work with our Unity SDK. Now the Cardboard SDK has been open sourced and Metal support is on the horizon we decided we can make the switch and transform our own SDK to use the Cardboard SDK. You will see me promoting some other changes here as well :)