Merge pull request #1 from cabanier/main

Initial submission

README.md

@@ -1,5 +1,77 @@
+# WebXR Body Tracking
 
-# Specification 'body-tracking'
+The [WebXR Body Tracking Specification][this-spec] adds body tracking support in WebXR.
+Feature lead is Rik Cabanier ([@cabanier](https://github.com/cabanier)). 
 
-This is the repository for body-tracking. You're welcome to contribute! Let's make the Web rock our socks
-off!
+## Taking Part
+
+1. Read the [code of conduct][CoC]
+2. See if your issue is being discussed in the [issues][this-spec], or if your idea is being discussed in the [proposals repo][cgproposals].
+3. We will be publishing the minutes from the bi-weekly calls.
+4. You can also join the working group to participate in these discussions.
+
+## Specifications
+
+* [WebXR Body Tracking][this-spec]: Body Tracking support in WebXR
+* [Explainer](explainer.md)
+
+
+### Related specifications
+* [WebXR Device API - Level 1][webxrspec]: Main specification for JavaScript API for accessing VR and AR devices, including sensors and head-mounted displays.
+
+See also [list of all specifications with detailed status in Working Group and Community Group](https://www.w3.org/immersive-web/list_spec.html). 
+
+## Relevant Links
+
+* [Immersive Web Community Group][webxrcg]
+* [Immersive Web Working Group Charter][wgcharter]
+* [Originating proposal](https://github.com/immersive-web/proposals/issues/48)
+
+## Communication
+
+* [Immersive Web Working Group][webxrwg]
+* [Immersive Web Community Group][webxrcg]
+* [GitHub issues list](https://github.com/immersive-web/layers/issues)
+* [`public-immersive-web` mailing list][publiclist]
+
+## Maintainers
+
+To generate the spec document (`index.html`) from the `index.bs` [Bikeshed][bikeshed] document:
+
+```sh
+bikeshed spec
+```
+
+## Tests
+
+For normative changes, a corresponding
+[web-platform-tests][wpt] PR is highly appreciated. Typically,
+both PRs will be merged at the same time. Note that a test change that contradicts the spec should
+not be merged before the corresponding spec change. If testing is not practical, please explain why
+and if appropriate [file a web-platform-tests issue][wptissue]
+to follow up later. Add the `type:untestable` or `type:missing-coverage` label as appropriate.
+
+
+## License
+
+Per the [`LICENSE.md`](LICENSE.md) file:
+
+> All documents in this Repository are licensed by contributors under the [W3C Software and Document License](https://www.w3.org/Consortium/Legal/copyright-software).
+
+# Summary
+
+For more information about this proposal, please read the [explainer](explainer.md) and issues/PRs.
+
+<!-- Links -->
+[this-spec]: https://immersive-web.github.io/webxr-body-tracking
+[CoC]: https://immersive-web.github.io/homepage/code-of-conduct.html
+[webxrwg]: https://w3.org/immersive-web
+[cgproposals]: https://github.com/immersive-web/proposals
+[webxrspec]: https://immersive-web.github.io/webxr/
+[webxrcg]: https://www.w3.org/community/immersive-web/
+[wgcharter]: https://www.w3.org/2020/05/immersive-Web-wg-charter.html
+[webxrref]: https://immersive-web.github.io/webxr-reference/
+[publiclist]: https://lists.w3.org/Archives/Public/public-immersive-web-wg/
+[bikeshed]: https://github.com/tabatkins/bikeshed
+[wpt]: https://github.com/web-platform-tests/wpt
+[wptissue]: https://github.com/web-platform-tests/wpt/issues/new

explainer.md

@@ -0,0 +1,253 @@
+# WebXR Device API - Body Tracking
+
+This document describes a design giving developers access to body-tracking XR systems, building on top of the [WebXR device API](https://immersive-web.github.io/webxr/)
+
+## Use cases and scope
+
+This API primarily exposes the joints of a person's body. It can be used to render an avatar in VR scenarios. It does not provide access to a full hbodynd mesh.
+
+## Accessing this API
+
+This API will only be accessible if a `body-tracking"` [XR feature](https://immersive-web.github.io/webxr/#feature-dependencies) is requested.
+
+This API presents itself as an additional field on `XRFrame`, `body`. The `body` attribute will be non-null if the XR device supports hand tracking and the feature has been requested.
+
+```js
+navigator.xr.requestSession({optionalFeatures: ["body-tracking"]}).then(...);
+
+function renderFrame(session, frame) {
+ // ...
+
+ if (frame.body.hand) {
+ // render a body
+ }
+ }
+}
+
+
+```
+
+## Joints
+
+Each body is made up many bones, connected by _joints_. We name them with their connected bone, for example `index-finger-phalanx-distal` is the joint closer to the wrist connected to the distal phalanx bone of the index finger. The `*-tip` "joints" locate the tips of the fingers. The `wrist` joint is located at the composite joint between the wrist and forearm.
+
+The joint spaces can be accessed via `XRBody.get()`, for example to access the leftmiddle knuckle joint one would use:
+
+```js
+let joint = inputSource.hand.get("left-middle-finger-phalanx-distal");
+```
+
+All devices which support hand tracking will support or emulate all joints, so this method will always return a valid object as long as it is supplied with a valid joint name. If a joint is supported but not currently being tracked, the getter will still produce the `XRBodySpace`, but it will return `null` when run through `getPose` (etc).
+
+Each joint space is an `XRSpace`, with its `-Y` direction pointing perpendicular to the skin, outwards from the palm, and `-Z` direction pointing along their associated bone, away from the wrist. This space will return null poses when the joint loses tracking.
+
+For `*-tip` joints where there is no associated bone, the `-Z` direction is the same as that for the associated `distal` joint, i.e. the direction is along that of the previous bone.
+
+<!---
+## Displaying hand models using this API
+
+Ideally, most of this will be handled by an external library or the framework being used by the user.
+
+A simple skeleton can be displayed as follows:
+
+```js
+const orderedJoints = [
+ ["thumb-metacarpal", "thumb-phalanx-proximal", "thumb-phalanx-distal", "thumb-tip"],
+ ["index-finger-metacarpal", "index-finger-phalanx-proximal", "index-finger-phalanx-intermediate", "index-finger-phalanx-distal", "index-finger-tip"]
+ ["middle-finger-metacarpal", "middle-finger-phalanx-proximal", "middle-finger-phalanx-intermediate", "middle-finger-phalanx-distal", "middle-finger-tip"]
+ ["ring-finger-metacarpal", "ring-finger-phalanx-proximal", "ring-finger-phalanx-intermediate", "ring-finger-phalanx-distal", "ring-finger-tip"]
+ ["pinky-finger-metacarpal", "pinky-finger-phalanx-proximal", "pinky-finger-phalanx-intermediate", "pinky-finger-phalanx-distal", "pinky-finger-tip"]
+];
+
+function renderSkeleton(inputSource, frame, renderer) {
+ let wrist = inputSource.hand.get("wrist");
+ if (!wrist) {
+ // this code is written to assume that the wrist joint is exposed
+ return;
+ }
+ let wristPose = frame.getJointPose(wrist, renderer.referenceSpace);
+ renderer.drawSphere(frame, wristPose.transform, wristPose.radius);
+ for (finger of orderedJoints) {
+ let previous = wristPose;
+ for (joint of finger) {
+ let joint = inputSource.hand.get(joint);
+ if (joint) {
+ let pose = frame.getJointPose(joint, renderer.referenceSpace);
+ drawSphere(frame, pose.transform, pose.radius);
+ drawCylinder(frame,
+ /* from */ previous.transform,
+ /* to */ pose.transform,
+ /* specify a thinner radius */ pose.radius / 3);
+ previous = pose;
+ }
+ }
+ }
+}
+```
+
+## Hand interaction using this API
+
+It's useful to be able to use individual fingers for interacting with objects. For example, it's possible to have fingers interacting with spherical buttons in space:
+
+```js
+const buttons = [
+ {position: [1, 0, 0, 1], radius: 0.1, pressed: false,
+ onpress: function() { ... }, onrelease: function() { ... }},
+ // ...
+];
+
+function checkInteraction(button, inputSource, frame, renderer) {
+ let tip = frame.getPose(inputSource.hand.get("index-finger-tip"), renderer.referenceSpace);
+ let distance = calculateDistance(tip.transform.position, button.position);
+ if (distance < button.radius) {
+ if (!button.pressed) {
+ button.pressed = true;
+ button.onpress();
+ }
+ } else {
+ if (button.pressed) {
+ button.pressed = false;
+ button.onrelease();
+ }
+ }
+}
+
+function onFrame(frame, renderer) {
+ // ...
+ for (button of buttons) {
+ for (inputSource of frame.session.inputSources) {
+ checkInteraction(button, inputSource, frame, renderer);
+ }
+ }
+}
+
+```
+
+## Gesture detection using this API
+
+One can do gesture detection using the position and orientation values of the various fingers. This can get pretty complicated and stateful, but a straightforward example below would be simplistically detecting that the user has made a fist gesture:
+
+```js
+function checkFistGesture(inputSource, frame, renderer) {
+ for (finger of [["index-finger-tip", "index-finger-metacarpal"],
+ ["middle-finger-tip", "middle-finger-metacarpal"],
+ ["ring-finger-tip", "ring-finger-metacarpal"],
+ ["pinky-finger-tip", "pinky-finger-metacarpal"]]) {
+ let tip = finger[0];
+ let metacarpal = finger[1];
+ let tipPose = frame.getPose(inputSource.hand.get(tip), renderer.referenceSpace);
+ let metacarpalPose = frame.getPose(inputSource.hand.get(metacarpal), renderer.referenceSpace)
+ if (calculateDistance(tipPose.position, metacarpalPose.position) > minimumDistance ||
+ !checkOrientation(tipPose.orientation, metacarpalPose.orientation)) {
+ return false
+ }
+ }
+ return true;
+}
+
+function checkOrientation(tipOrientation, metacarpalOrientation) {
+ let tipDirection = applyOrientation(tipOrientation, [0, 0, -1]); // -Z axis of tip
+ let palmDirection = applyOrientation(metacarpalOrientation, [0, -1, 0]) // -Y axis of metacarpal
+
+ if (1 - dotProduct(tipDirection, palmDirection) < minimumDeviation) {
+ return true;
+ } else {
+ return false;
+ }
+}
+```
+
+## Efficiently obtaining hand poses
+
+Each use of `getPose()` allocates one short-lived `XRPose` object, one `XRRigidTransform` object, and at least one `DOMPointReadOnly` or `Float32Array`. For 25 joints per hand and two hands, this is 150-250 objects created per frame. This can have noticeable performance implications, especially around garbage collection.
+
+To avoid that, we provide a `fillPoses()` (and `fillJointRadii`) API which can be used to efficiently obtain all the transforms for a hand at once.
+
+```js
+let poses1 = new Float32Array(16 * 25);
+let radii1 = new Float32Array(25);
+function onFrame(frame, renderer) {
+ let hand1 = frame.session.inputSources[0].hand;
+ frame.fillPoses(hand1.values(), renderer.referenceSpace, poses1);
+ frame.fillJointRadii(hand1.values(), radii1);
+ renderer.drawHand(poses1, radii1);
+ // do something similar for second hand
+}
+```
+
+## Privacy and Security Considerations
+
+The concept of exposing hand input could pose a risk to users’ privacy. For example, data produced by some hand-tracking systems could potentially enable sites to infer users’ gestural behaviors or approximate hand size, make it apparent to sites that a user is missing fingers or parts of fingers, or allow detection of tremors or other medical conditions.
+
+Implementations are required to employ strategies to mitigate these risks, such as:
+- Reducing the precision and sampling rate of data
+- Adding noise or rounding data
+- Return the same hand geometry/size for all users
+- Emulating values for joints if the implementation isn’t capable of detecting them or the user does not have them.
+
+This specification requires implementations to include sufficient mitigations to protect users’ privacy.
+
+## Appendix: Proposed IDL
+
+```webidl
+partial interface XRInputSource {
+ readonly attribute XRHand? hand;
+}
+
+partial interface XRFrame {
+ XRJointPose? getJointPose(XRJointSpace joint, XRSpace relativeTo);
+ boolean fillJointRadii(sequence<XRJointSpace> jointSpaces, Float32Array radii);
+ boolean fillPoses(sequence<XRSpace> spaces, XRSpace baseSpace, Float32Array transforms);
+}
+
+interface XRJointPose: XRPose {
+ readonly attribute float? radius;
+}
+
+interface XRJointSpace: XRSpace {
+ readonly attribute XRHandJoint jointName;
+}
+
+enum XRHandJoint {
+ "wrist",
+
+ "thumb-metacarpal",
+ "thumb-phalanx-proximal",
+ "thumb-phalanx-distal",
+ "thumb-tip",
+
+ "index-finger-metacarpal",
+ "index-finger-phalanx-proximal",
+ "index-finger-phalanx-intermediate",
+ "index-finger-phalanx-distal",
+ "index-finger-tip",
+
+ "middle-finger-metacarpal",
+ "middle-finger-phalanx-proximal",
+ "middle-finger-phalanx-intermediate",
+ "middle-finger-phalanx-distal",
+ "middle-finger-tip",
+
+ "ring-finger-metacarpal",
+ "ring-finger-phalanx-proximal",
+ "ring-finger-phalanx-intermediate",
+ "ring-finger-phalanx-distal",
+ "ring-finger-tip",
+
+ "pinky-finger-metacarpal",
+ "pinky-finger-phalanx-proximal",
+ "pinky-finger-phalanx-intermediate",
+ "pinky-finger-phalanx-distal",
+ "pinky-finger-tip"
+};
+
+interface XRHand {
+ iterable<XRHandJoint, XRJointSpace>;
+
+ readonly attribute unsigned long size;
+ XRJointSpace get(XRHandJoint key);
+};
+
+
+```
+--->