Animation support checked in!

[vsg-dev]

Hi All,

Over the past month I have been working on rigid body and skinning animation support for the VulkanSceneGraph and am delighted to announce it's all now merged with VulkanSceneGraph, vsgXchange and vsgExample master. The changes are:

• VulkanSceneGraph - 121 commits, 135 files changed!
• vsgXchange
• vsgExamples

The changes to the VulkanSceneGraph encompass new animation classes that can be found in include/vsg/animation and far broader changes that add copy constructor and cloning support to the main scene graph objects, which was required in order to implement clone of models that are loaded with animations.

The changes to vsgXchange are focused on the vsgXchange::assimp loader, where the assimp animation data structures are parsed and converted to the appropriate VSG animation classes.

The changes to vsgExamples related to add skinning support to the standard.vert shader and the associated Phong, PBR and flat shaded ShaderSets, and the addition of a new vsganimation example.

The new animations classes are:

• vsg::AnimationGroup - scene graph node that decorates a subgraph/model that contains animations, and has a list of Animations.
• vsg::Animation - representations a single animation made up of one or more vsg::AnimationSamplers.
• vsg::TransformSampler - animation sampler that samples position, rotation and scale key frame data and assigns the result to a specified transform node/joint.
• vsg::JointSampler - animation sampler traverses a specified joints hierarchy and assigns accumulates transforms to a jointsMatrices array that can be assigned to a vsg::DescriptorBuffer for rendering. JointSampler is used in conjunction with TransformSampler, with the joint positions being updated by the TransformSampler and the final joint matrices accumulated and passed to the GPU via the JointSampler and the VSG's dynamic data transfer support.
• vsg::MorphSampler - not yet functional, but will support geometry morphing between key frames.

Users will be able to set up the Animation classes themselves, but I expect most users will leverage the the vsgXchange::assimp loader to load 3rd party models which have animation built in.

More info/examples to follow soon.

[lufriem]

This is most welcome, thanks!

[robertosfield]

My goal for animation support has been to provide a clean and easy to use set of classes that enables users to load models with animations and clone those models as many times as they require whilst retaining individual control over animations. Whilst integrating fully with the PBR, Phong and Flat ShaderSets that comes built-in into the core VSG. Full support for cascaded shadow maps and handling of large coordinate systems is provided out of the box.

For the examples I'll go into below I'll be using the official Khronos glTF assets:

https://github.com/KhronosGroup/glTF-Sample-Assets.git

To see your first animated object try the new vsganimation example:

vsganimation glTF-Sample-Assets/Models/BrainStem/glTF/BrainStem.gltf

What you see is a manic character dancing away:

This example loads the models from the command line, places them on a grid and enable shadow mapping as well as provide a few keyboard controls.

To load a model and find all the animations in that model you can use the standard vsg::read(..) methods and the new vsg::FindAnimations visitor:

    // standard setup of options for reading from vsgXchange and finding files, and sharing objects within and between loads
    auto options = vsg::Options::create();
    options->sharedObjects = vsg::SharedObjects::create();
    options->fileCache = vsg::getEnv("VSG_FILE_CACHE");
    options->paths = vsg::getEnvPaths("VSG_FILE_PATH");
    options->add(vsgXchange::all::create());

    // load the model
    auto model = vsg::read_cast<vsg::Node>("glTF-Sample-Assets/Models/BrainStem/glTF/BrainStem.gltf", options);
    if (!model) return 1; // should do some proper file not loaded handling, but removed for brevity

    // find animations
    auto animationGroups = vsg::visit<vsg::FindAnimations>(model);

To play the first animation of each of animation groups found simply iterator through animation groups and call vsg::AnimationManager::play(animation) on the first animation available. The AnimationManager is assigned by default to the vsg::Viewer, so no need to add anything, all you need to do is tell the animationManager to play your animation:

    for(auto ag : animationGroups)
    {
        if (!ag->animations.empty()) viewer->animationManager->play(ag->animations.front());
    }

This works the same for rigid body models as well as ones that having skinning like the above BrainStem.gltf model, there's no additional set up required.

[robertosfield]

The glTF sample set has a number of models with rigid body and skinning animations, and the vsganimation example is able to load them all together and then them all at the same time:

cd glTF-Sample-Assets/Models
vsganimation BrainStem/glTF/BrainStem.gltf CesiumMan/glTF/CesiumMan.gltf CesiumMilkTruck/glTF/CesiumMilkTruck.gltf Fox/glTF/Fox.gltf SimpleSkin/glTF/SimpleSkin.gltf RiggedFigure/glTF/RiggedFigure.gltf BoxAnimated/glTF/BoxAnimated.gltf VirtualCity/glTF/VirtualCity.gltf RiggedSimple/glTF/RiggedSimple.gltf

The example has an event handler that provides the follow key controls:

• p play a single animation and then move the iterator to the next animation on, so next time you press p another animation will play.
• s stop all animations from playing.
• a start the first animation in each of the available animation groups.
• < lower the Animation::speed ratio by 0.25, the default speed is 1.0 for full animation speed, 0 stops, negative goes backwards.
• > increase the Animation::speed ratio by 0.25.

The example records compile, frame rate and update time stats so we can see how quickly the animations take to run, for the above example run, on my Kubuntu 22.04, AMD 5700G + Geforce 2060 I see:

Compile time : 47.7984 ms
Average frame rate = 60.0035
Average update time = 0.0354873 ms

[robertosfield]

A key part of the animation work was making sure if user have an animation model that they want to instance multiple times in a scene they can still control the animations independently, whilst sharing as much as the scene graph between each of the load instances. This turns out to be a non trivial task and over half commits and most of the file changed for this work related to extending the scene graph so that it can clone models in a flexible and efficient way.

To illustrate this sharing in action I have run vsganimation with the -n <num_copies> command line option, and -o to output to file, and --no-base to prevent it from creating the base the animation runs:

vsganimation --no-base BrainStem/glTF-Binary/BrainStem.glb -o one.vsgb
vsganimation --no-base BrainStem/glTF-Binary/BrainStem.glb -n 10 -o ten.vsgb
vsganimation --no-base BrainStem/glTF-Binary/BrainStem.glb -n 1000 -o onethousand.vsgb

The sizes of the result file are:

| BrainStem.glb | 3194848 |
| one.vsgb | 3175334 |
| ten.vsgb | 3324113 |
| onethousand.vsgb | 19689803 |

This is what one thousand BrainStem models looks like:

vsganimation BrainStem/glTF-Binary/BrainStem.glb -n 1000

One thousand BrainStem models all animating and with cascaded shadow maps does slow my machine a bit, adding --IMMEDIATE to the command line to turn off vsync the results I see are:

Compile time : 148.24 ms
Average frame rate = 14.5152
Average update time = 3.27299 ms

Switching off shadow mapping with --sm 0 bumps the frame rate to 50fps. This is a relatively large model with a thousand copes all updating joints and transferring the joint matrices to GPU memory and running vertex shader that does the accumulation of the skiing matrices.

If I use a simpler model like CesiumMan.glb and copy it a thousand times I get 184fps with cascaded shadow maps on, and 291fps with the shadow mapping off. The CPU cost of skinning is higher than that of simple rigid body animation, for instance a thousand CesiumMilkTruck.gltf just require 0.18ms for all four thousand wheels.

The magic that makes this possible are a few new scene graphs features:

• new virtual vsg::Object::clone(const vsg::CopyOp&) method that subclasses implement to clone themselves and their child objects. I then implemented copy constructors and clone(..) methods is a wide range of scene graph objects to ensure they get cloned correctly when required.
• new vsg::CopyOp and vsg::Duplicate classes to provide a mechanism for specify which child objects should be duplicated and to reuse any duplicates which are then made.
• new vsg::FindDynamicObjects class for finding parts of scene graph that should treated as dynamic and must have their own unique copy when cloning the scene graph - for instance any vsg::Data which as properities.dataVariance which is not set to STATIC_DATA, any vsg::AnimationGroup, vsg::Animation or vsg::AnimationSampler.
• new vsg::PropogateDynamicObjects class that traverses the scene graph and marks any objects as dynamic if they have any child objects that are marked by dynamic by the FindDynamicObjects visitor and propogates this up the chain to root of the subgraph. This mechanism ensures that the DescriptorBuffer that contains a dynamically marker jointMatrices array is also marked as dynamic, and all DescriptorSet that use that Descriptor are marked as dynamic and so on all the way up the subgraph.
• vsg:Options now has findDynamicObjects and propogateDynamicObjects members that default to the above visitors but can be assigned new implementations if users want to customize this behavior.
• [vsg::ShareObject](https://github.com/vsg-dev/VulkanSceneGraph/blob/master/include/vsg/utils/SharedObjects.h#L33 now :LoadedObject has a new set<const Object*> dynamicObjects member for specify which objects are dynamic, this set is populated by the find and propagate visitors.
• Finally vsg::read(..) now automatically utilizes the LoadedObjects::dynamicObjects set decide when subsequently loading a model that is already cached in the SharedObjects, whether the model should be cloned and if it does need cloning then what parts of the scene graph should be cloned, then clones the model and wires up all the duplicates within that new model and sharing all the parts that aren't declared as dynamic.

This scheme has lots of parts to it but has solve a what turned out to be surprisingly complicated problem, but this is all in aid of making it very straightforward of developers to just load the same model files as many times as they want, with or without knowing they have animations and without having to do anything themselves to facilitate the correct and efficient sharing of parts of the loaded scene graph that can be shared and duplicating the rest.

For instance, if you've assign a SharedObjects to Options as I did in the original code snippet earlier in this forum thread and then load the same file multiple times, you'll need with two models that have all their animation parts duplicated and the rest shared:

auto model_one = vsg::read_cast<vsg::Node>("CesiumMan.glb", options);
auto model_two = vsg::read_cast<vsg::Node>("CesiumMan.glb", options);

These types of calls would be typically done at various points in the lifetime of an application, rather conveniently together like this, but as lone as your use the same SharedObjects instance between the vsg::read<> calls then the same clone/duplicate process will be automatically be invoked.

Those familiar with the OpenSceneGraph project may know about the osg::Object::clone(const osg::CopyOp&) method and the presence of all these copy constructors just the VSG now has, but the osg::CopyOp is a very different beast - it holds a bit mask that tells copy constructors which types of objects to share and which to duplicate, rather than a set of objects that should be duplicated. There is also no mechanism for sharing duplicates in the same way the vsg::CopyOp/vsg::Duplicate enable, The OSG takes a woodsman approach of hacking away at the tree while the VSG takes a surgeon approach with the end results of cloning only the parts that are essential to clone and sharing the rest, something that the cruder OSG can't do which leads to far more duplication of data/objects.

Those familiar with the OSG animation support will also know about the animation callbacks and that hardware rigging has to be setup, and that you'll need your own custom shaders especially if you want to add shadows. The VSG approach is far simpler for the end users - just use the vsg::FindAnimations visitor to find the animations, and animationManager->play(...) to play them, everything else just works out of the box exactly the same way as for static models.

[robertosfield]

Another key part of getting animations to work seamlessly in VSG applications is the ability to handle whole earth databases where animated models are placed on paged databases, and have to rendering with jitter to the movement, placement, lighting or shadows.

As the rest of the VSG is built to handle large coordinates the animation work didn't have to do anything, it just works out of the box, to test this out I added support for loading a whole earth paged database and placing the loaded animated models at specified location. The follow illustrates adding a 1000 CesiumMan models besides Buckingham Place:

vsganimation CesiumMan/glTF/CesiumMan.gltf -n 1000 --earth models/openstreetmap.vsgt --location 51.501 -0.142 0 --scale 5  --sd 1000

Close up:

And for fun just a single CesiumMan but scaled up so much even Godzilla would crawl quietly back into the ocean..

vsganimation CesiumMan/glTF/CesiumMan.gltf  --earth models/openstreetmap.vsgt --location 51.501 -0.142 0 --scale 50000 --direction -1 1 0

[robertosfield]

While this is all fun, there is a problem with Assimp loading of skinned FBX models, here's what dance.fbx from mixamo.com:

vsganimation dance.fbx

Mixamo has support for download fbx and glb models so here's an example of the same model loaded in fbx to the left, and glb (gltF binary) to the right:

vsganimation Catwalk.fbx Catwalk.glb 

Looking the the Assimp issue tracker there are others who have seen similar problems with recent version of Assimp - I'm using v5.3.1. Further investigation is required.

[robertosfield]

I consider the animation work just checked in as good first step, but they is still more areas to develop w.r.t interface, implementation and breadth of features. Input from the community will really help refine and extend what we have so if you have ideas that feel free to chip in.

In no particular order, some of the possibilities for the core VSG and other libraries:

1. Support for step and cubic interpolations - the current implement uses linear interpolation
2. Refinement of skinning - current the code uses joint matrices and linear blending of joint matrices in the vertex shader which is crude.
3. Utilities for blending/transitions between animations
4. Implementation for geometry morphing
5. A dedicated glTF loader rather than relying upon Assimp
6. AnimationSampler implementation for animating of uniforms
7. Particle system animations/effects like fire, rain, rain etc.
8. ArrayState support for skinning so the we can do CPU geometry tests like intersections
9. Resolving problems like the Assimp/FBX joint matrices issue

---

I the short term I would appreciate testing of the animation functionality out in the community, once it looks it like VSG master is compiling and running cleanly across platforms I can make a 1.1.2 development release to provide a tag that includes the animation functionality.

[robertosfield]

I have just tackled the unification of camera path handlers with the new vsg::Animation, changes to VSG, osg2vsg and vsgExamples have now been merged with the respective masters:

• Changes to VulkanSceneGraph
• Changes to osg2vsg
• Changes to vsgExamples

Previously I have implemented vsg::AnimationPath and vsg::AnimationPathHandler that mirrored the osg::AnimationPath and osg::AnimationPathHandler found in the OpenSceneGraph, I did this to be able to do direct performance comparisons between the two scene graphs as I could share the same .path generated from osgviewer.

With the introduction of the vsg::Animation functionality last month we now had additional of implementation animation paths for vsg::MatrixTransform, but this wasn't yet wired up for vsg::Camera/vsg::LookAt so we had two divergent solutions for essentially the same task, with the old vsg::AnimationPath far less capable than new vsg::Animation/TransformSampler classes.

It's been on my TODO list replace vsg::AnimationPath/vsg::AnimationPathHandler/vsg::RecordAnimationPath classes with use of vsg::Animation instead and yesterday I began that work and have just wrapped it up. I updated osg2vsg to now create vsg::Animation/TransformSampler objects when reading an OSG's .path file, and updated all the examples as well.

I have removed vsg::AnimationPath, vsg::AnimationPathHandler and vsg::RecordAnimationPath classes completely with the latest check-in, the vsg::AnimationPath is replaced by vsg::Animation/TransformSampler/TransformKeyframes usage, and the two event handlers are replaced by vsg::CameraAnimation event handler which adds ability to record new camera paths and same them to file, as well as play camera paths.

The key controls for the vsg::CameraAnimation event handler are:

• r - first press starts recording the camera path, second press stops recording and saves to file (defaults to saved_animation.vsgt)
• p - first press plays the camera path, second press stops the playback.