CORPUS is a multisensor data fusion framework for sensors tracking the human body or parts of it. In particular the framework tries to satisfy the following requirements.
The framework is able to automatically fuse and filter the data delivered by the sensors. The result is a uniform body model composed of joints. It is possible to access the position and orientation of a joint. Thus, the developer doesn't need to deal with the specific details of the sensors or their API anymore. The only requirement is a implemented wrapper for the sensor, but this only needs to be done once.
The framework is designed as a client-server structure. The client application can be connected to the server via a RESTful webservice or a WebSocket connection. The body model is then sent to the client in form of a JSON string.
The framework is highly extensible and adaptable. The fusing process is executed by a class extending the Fuser class. The framework provides example implementation of this class using general fusing strategies. However, it is possible to write custom implementations to adapt the fusing process. The same is true for the Filter class responsible for the filter process. Wrappers are also easy implementable by extending the Sensor class.
Furthermore, the framework is configured through a configuration file. This enables the user to change the update rate, the base URI, the granularity of the body model and more.
The update chain is designed to support multiple sensors. Only a wrapper for a sensor needs to be added to the framework trough the configuration file.
The framework is written in java and uses the Grizzly Project to implement the server. The REST Webservice is realized with the help of Jersey. Currently wrappers are available for the Microsoft Kinect v1, the Leap Motion, the Oculus Rift Dev Kit 1 and Dev Kit 2 and the Myo. To communicate with the sensors the libraries Kinect-for-Java, jovr and myo-java are used.
The provided project is a eclipse project. To manage the dependencies, Maven is used. Therefore, no additional libraries should be needed. The framework was tested with Windows 7 64 bit. This repository mainly includes native libraries for 64 bit Windows systems. Other libraries may be needed for different systems.
The build process is also managed by maven and can be executed by the following command:
mvn clean installThe folder target then contains a executable java jar file.
In the context of the framework the human is represented in form of joints. The joints are defined in a hierarchical way. That means each joint has a parent. The connection between a joint and its parent can be interpreted as a bone, but is not explicitly stated in the body representation. The position and orientation of a joint is relative to the parent. The orientation describes how the coordinate system of the parent has to be rotated so that the bone created by the joint and his parent lies on the y-axis. If there is no parent, the joint is a root joint and the global coordinate system is considered. The position of a joint uses the coordinate system of the parent. The orientations are represented as quaternions.
The body model is divided into 3 parts - torso, hands and feet. For each of the three parts the user can set the needed granularity. Namely these are NONE, SPARSE and COMPLEX. The following table describes the granularity levels in detail.
| Body Part | Granularity | Contained Joints |
|---|---|---|
| Torso | NONE | none |
| SPARSE | middle, left and right hip, knees, middle, left and right shoulder, elbow, head | |
| COMPLEX | SPARSE + middle spine, neck | |
| Hands | NONE | none |
| SPARSE | wrists, center of the hands | |
| COMPLEX | SPARSE + every moveable finger joint | |
| Feet | NONE | none |
| SPARSE | ankles, center of the feet | |
| COMPLEX | SPARSE + every moveable toe joint |
Joints which are not detected by a sensor are set to their default position after a specified period of time. The following picture shows a body model with full complexity. All joints are in their default position. The center hip serves as the root joint due to its central position within the human body.
If a client application requests the current model, JSON data is generated. The following lines show a part of an exemplary JSON representation of the body model.
[ { "jointType": "SPINE_BASE",
"relativePosition": {
"x": 0, "y": 0, "z": 300},
"relativeOrientation": {
"w": 0, "x": 0, "y": 1, "z": 0},
"absolutePosition": {
"x": 0, "y": 0, "z": 300},
"absoluteOrientation": {
"w": 0, "x": 0, "y": 1, "z": 0},
"positionConfidence": 0,
"orientationConfidence": 0,
"positionTracked": false,
"orientationTracked": false,
"positionTimestamp": 1420629906784,
"orientationTimestamp": 1420629906784,
"children": [
{ "jointType": "HIP_RIGHT",
...},
...] } ]The model is returned as a list of root joints (joints without a father). In this sample case the only root joint in the current model is the joint with the type SPINE_BASE. The relative and absolute values are the same as there is no father for this joint. The confidence values are set by the sensor wrapper and may be manipulated by the used Fuser. In this case the values are set to zero because the joint is not tracked as stated in the following line. The timestamp states the point in time when the joint was created (as UTC milliseconds from the epoch). The children are serialized in the same way as described for the root joint. It is not mandatory that all of the attributes shown are transferred. It is also possible to define in the REST request which attributes are required. The following REST requests are provided by the framework:
| Method | Arguments | Description |
|---|---|---|
| fullHierarchicalModel | none | Sends the model in hierarchical form. |
| fullListModel | none | Sends the model in form of a list. |
| customModel | type, field | Sends all joints contained by the body model, but allows to choose the typ of the model (list orhierarchical) and the fields that will be serialized. |
| singleJoints | jointType, field | Sends the joints whose types are defined by the jointType attribute. The serialized fields are again defined by the field argument. |
| sensors | none | Sends all active sensors. |
| callSensorMethod | sensorId, methodName, param | Enables the user to call a sensor specific method. |
All methods besides the last one are realized as REST GET opertaions. The following request for example asks for the joint with the type HEAD and the fields jointType, absolutePosition and absoluteOrientation:
http://localhost:8080/corpus/singleJoint?jointType=HEAD&field=jointType&field=absolutePosition&field=absoluteOrientationThe last method enables the user to call a sensor specific method. This method has, of course, to be implemented in the corresponding wrapper. The method should be used only if necessary because it contradicts the sensor abstraction targeted by the framework.
The image shows how a client request is handled, including the starting process of the server and the update process of the model. The Controller starts the server and handles client requests. The update process is independent of any request and is run by the SceneController multiple times per second. The specific update rate can be adopted to the requirements of the client application. The update process includes the call of the Fuser which collects the data from the sensors by itself and fuses the received data in a specific manner. How the data is fused in detail depends on the chosen implementation of the Fuser. The fused data is then passed to the Filter. How the data is filtered again depends on the implementation of the Filter. The SceneController then can generate a JSON string containing the current model which is returned by the Controller to the client. The important components of the framework are explained in detail in the following section.
The Controller is the main component of the framework. It starts the server which includes the parsing of the configuration file. The configuration file can, for example, be used to customize the body model or to change the update rate of the model. The Controller also handles the client requests and initializes the SceneController.
The SceneController runs in its own thread and contains and updates the current Scene. A Scene contains the positions and orientations of the sensors and the joints at a specific point in time. In the configuration file the user has to define an initial Scene. This Scene enables the user to define the used sensors and the body model. The sensors are also placed in the Scene because their positions and orientations can change during the runtime of the framework - for example, if a sensor is placed relative to a joint. The SceneController also generates the JSON representation of the current Scene and contains a history of elapsed Scenes. This history can be used during the filter process or by client applications. Currently only a single Scene is supported.
The SceneNode is an abstract class and represents objects that can be placed in the Scene. A scene node can have a parent. The fields relativePosition and relativeOrientation contain relative values to this parent as explained in the chapter describing the body model. The fields absolutePosition and absoluteOrientation accordingly contain the absolute values. The setter and getter for the relative and absolute values are designed to ensure consistence. If a relative value is updated, the absolute value will be updated as well if necessary and vice versa.
A Joint extends the SceneNode class and represents a brick of the body model. The type of the joint is indicated by the JointType. In addition to the fields of the SceneNode class a Joint contains information about the tracking state and the confidence of the tracked values. It also contains a default position and orientation.
The framework provides an abstract Sensor class which can be implemented for any sensor providing positional or orientational data of human body regions. A Sensor extends the SceneNode class. A Sensor serves two purposes in the context of the framework. Firstly, it is a wrapper communicating with the actual sensor and processing the delivered data. Secondly, a Sensor (the wrapper) is also placed in the Scene. Therefore, it also represents the actual sensor in the context of the framework.
One Fuser per Scene is responsible for merging the data delivered by multiple sensors. It is possible to create a custom implementation. In the following, the fusing process is described on the basis of the ConfidenceDominanceFuser provided by the framework. The Fuser collects the newest data of the sensors and expects that the confidence for the position and orientation is set. On the basis of these values, the Fuser now determines which data will be taken. If no sensor provides data for a specific joint, the data of the last Scene will be preserved or the position of the joint is reset to its default position - if there was no new data for a specific period of time. If only one sensor provides new data for a joint, the data is taken and if multiple sensors provide data for a joint, the data with the highest confidence value is taken. The confidence value of the final joint is the one of the chosen joint. This procedure is repeated for every joint in the model. Thus, data for joints not contained in the model is ignored.
The Scene with the fused positions and orientations is forwarded to the Filter. The Filter can be implemented by the user and there is no restriction on how the data is filtered. Inverse kinematics could be used to ensure that the position of the joints correlate with the potential moving space of the human body. Likewise, a Filter could smooth the data to reduce the noise. Exemplary, a double exponential smoothing filter is used in this framework.
To support upcoming sensors, it is important to make the integration of new sensors as easy as possible. The framework allows this by simply extending the abstract Sensor class. The Sensor class contains the field currentData representing a Map which can be set with the help of the method setCurrentData. The keys of the Map are JointTypes and the values are Joints. This representation simplifies the fusion and filter process. Hence, the job of the wrapper is to map the collected data to the Joints supported by the framework. It is possible to set the relative values, but then the parents are expected, too. The confidence values should be set as well if a Fuser expecting these values is used.
There are three ways to get the data from the actual sensor. Firstly, it is possible to directly request the new data from the sensor when needed. Once in the update cycle the getCurrentData method is called for every wrapper. This method automatically calls the function updateCurrentData. it has to be implemented by the user and can be used to get the current data from the sensor. Secondly, a lot of devices work with an event based system, where new sensor data triggers an event to notify interested listeners. If a custom sensor wrapper uses this kind of update process, the listener should set the current data map in response to an event. In this case the updateCurrentData method can be left empty. The last option is to use the run method. Every wrapper runs in its own thread. In the run method it is possible to implement a loop continuously polling the data from the sensor.
A event based system and continuously polling the data increases the scalability of the framework because the polling process is moved away from the update process. On the other hand, polling the data only when needed probably saves unnecessary calls if the update rate of the sensor is higher than the one of the framework.
The configuration file (config.xml) is an XML file enabling the user to customize the framework. The committed configuration file is commented and is therefore not further explained at this point.
The framework lists every possible joint type in the enum JointType. In the following the joint types will be explained.
The image (Source: Modified version of LadyofHats) visualizes the joints for the left hand used in the framework. The following table maps the joint to the corresponding JointType value. For the right hand LEFT has to be simply replaced by RIGHT.
| JointType | Number | Joint |
|---|---|---|
| WRIST_LEFT | 1 | left wrist |
| HAND_CENTER_LEFT | 2 | center of the left palm |
| left small finger | ||
| CMC_SMALL_FINGER_LEFT | 3 | carpometacarpal (CMC) joint |
| MCP_SMALL_FINGER_LEFT | 4 | metacarpophalangeal (MCP) joint |
| PIP_SMALL_FINGER_LEFT | 5 | proximal interphalangeal (PIP) joint |
| DIP_SMALL_FINGER_LEFT | 6 | distal interphalangeal (DIP) joint |
| BTIP_SMALL_FINGER_LEFT | 7 | bone tip (BTIP) |
| left ring finger | ||
| CMC_RING_FINGER_LEFT | 8 | carpometacarpal (CMC) joint |
| MCP_RING_FINGER_LEFT | 9 | metacarpophalangeal (MCP) joint |
| PIP_RING_FINGER_LEFT | 10 | proximal interphalangeal (PIP) joint |
| DIP_RING_FINGER_LEFT | 11 | distal interphalangeal (DIP) joint |
| BTIP_RING_FINGER_LEFT | 12 | bone tip (BTIP) |
| left middle finger | ||
| CMC_MIDDLE_FINGER_LEFT | 13 | carpometacarpal (CMC) joint |
| MCP_MIDDLE_FINGER_LEFT | 14 | metacarpophalangeal (MCP) joint |
| PIP_MIDDLE_FINGER_LEFT | 15 | proximal interphalangeal (PIP) joint |
| DIP_MIDDLE_FINGER_LEFT | 16 | distal interphalangeal (DIP) joint |
| BTIP_MIDDLE_FINGER_LEFT | 17 | bone tip (BTIP) |
| left index finger | ||
| CMC_INDEX_FINGER_LEFT | 18 | carpometacarpal (CMC) joint |
| MCP_INDEX_FINGER_LEFT | 19 | metacarpophalangeal (MCP) joint |
| PIP_INDEX_FINGER_LEFT | 20 | proximal interphalangeal (PIP) joint |
| DIP_INDEX_FINGER_LEFT | 21 | distal interphalangeal (DIP) joint |
| BTIP_INDEX_FINGER_LEFT | 22 | bone tip (BTIP) |
| left thumb | ||
| CMC_THUMB_LEFT | 23 | carpometacarpal (CMC) joint |
| MCP_THUMB_LEFT | 24 | metacarpophalangeal (MCP) joint |
| IP_THUMB_LEFT | 25 | interphalangeal (IP) joint |
| BTIP_THUMB_LEFT | 26 | bone tip (BTIP) |
The image (Source: Modified version of Tasha) visualizes the joints of the left foot used in the framework. The following table maps the joint to the corresponding JointType value. For the right foot LEFT has to be simply replaced by RIGHT.
| JointType | Number | Joint |
|---|---|---|
| ANKLE_LEFT | 1 | left ankle |
| FOOT_CENTER_LEFT | 2 | center of the left foot |
| HEEL_BONE_LEFT | 3 | heel bone of the left foot |
| left small toe | ||
| TMT_SMALL_TOE_LEFT | 4 | tarsometatarsal (TMT) joint |
| MCP_SMALL_TOE_LEFT | 5 | metacarpophalangeal (MCP) joint |
| PIP_SMALL_TOE_LEFT | 6 | proximal interphalangeal (PIP) joint |
| DIP_SMALL_TOE_LEFT | 7 | distal interphalangeal (DIP) joint |
| BTIP_SMALL_TOE_LEFT | 8 | bone tip (BTIP) |
| left ring toe | ||
| TMT_RING_TOE_LEFT | 9 | tarsometatarsal (TMT) joint |
| MCP_RING_TOE_LEFT | 10 | metacarpophalangeal (MCP) joint |
| PIP_RING_TOE_LEFT | 11 | proximal interphalangeal (PIP) joint |
| DIP_RING_TOE_LEFT | 12 | distal interphalangeal (DIP) joint |
| BTIP_RING_TOE_LEFT | 13 | bone tip (BTIP) |
| left middle toe | ||
| TMT_MIDDLE_TOE_LEFT | 14 | tarsometatarsal (TMT) joint |
| MCP_MIDDLE_TOE_LEFT | 15 | metacarpophalangeal (MCP) joint |
| PIP_MIDDLE_TOE_LEFT | 16 | proximal interphalangeal (PIP) joint |
| DIP_MIDDLE_TOE_LEFT | 17 | distal interphalangeal (DIP) joint |
| BTIP_MIDDLE_TOE_LEFT | 18 | bone tip (BTIP) |
| left long toe | ||
| TMT_LONG_TOE_LEFT | 19 | tarsometatarsal (TMT) joint |
| MCP_LONG_TOE_LEFT | 20 | metacarpophalangeal (MCP) joint |
| PIP_LONG_TOE_LEFT | 21 | proximal interphalangeal (PIP) joint |
| DIP_LONG_TOE_LEFT | 22 | distal interphalangeal (DIP) joint |
| BTIP_LONG_TOE_LEFT | 23 | bone tip (BTIP) |
| left big toe | ||
| TMT_BIG_TOE_LEFT | 24 | tarsometatarsal (TMT) joint |
| MTP_BIG_TOE_LEFT | 25 | metatarsophalangeal (MTP) joint |
| IP_BIG_TOE_LEFT | 26 | interphalangeal (IP) joint |
| BTIP_BIG_TOE_LEFT | 27 | bone tip (BTIP) |
The image (Source: Modified version of DIAGRAM PICTURE) visualizes the joints of the torso used in the framework. The following table maps the joint to the corresponding JointType value. The wrists and ankles don't belong to the torso in the context of the framework, but to visualize the transition points of the body parts they are shown again.
| JointType | Number | Joint |
|---|---|---|
| SPINE_BASE | 1 | center hip |
| SPINE_MID | 2 | middle of the spine |
| SPINE_SHOULDER | 3 | spine and shoulder concourse |
| NECK | 4 | center neck |
| HEAD | 5 | head |
| SHOULDER_RIGHT | 6 | right shoulder |
| ELBOW_RIGHT | 7 | right elbow |
| WRIST_RIGHT | 8 | right wrist |
| SHOULDER_LEFT | 9 | left shoulder |
| ELBOW_LEFT | 10 | left elbow |
| WRIST_LEFT | 11 | left wrist |
| HIP_RIGHT | 12 | right hip |
| KNEE_RIGHT | 13 | right knee |
| ANKLE_RIGHT | 14 | right ankle |
| HIP_LEFT | 15 | left hip |
| KNEE_LEFT | 16 | left knee |
| ANKLE_LEFT | 17 | left ankle |
There is a sensor-fusion library called Jester with similar goals like the here provided framework. Some good ideas where adopted to this framework.
Copyright 2015 Matthias Weise [email protected]
GNU Lesser General Public License (LGPL) version 3 or later.
http://www.gnu.org/licenses/lgpl.html
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.