Skip to content

Latest commit

 

History

History
87 lines (68 loc) · 5.8 KB

README.md

File metadata and controls

87 lines (68 loc) · 5.8 KB
Raw

Raveler

Raveler

Raveler is a Wwise plugin that runs RAVE models, enabling real-time timbre transfer via neural audio synthesis in a game audio setting.

Much of the plugin implementation is based on the RAVE VST project.

Demo video: https://youtu.be/5RfZWJp6-i4

Usage

Simply add a Raveler plugin instance anywhere in your effects chain, specify a model to load, and start playing around!

The plugin consists of 4 main categories of parameters:

  1. Model Performance
    • Latent Noise: Amount of random noise/jitter to add to all latent dimensions
    • Use Prior: Whether or not to use the prior, if available in the model
    • Prior Temperature: The "temperature" to use when sampling the prior (only applies if we're using the prior)
    • Output Width: Amount of sampling differences between output channels
    • Output Dry/Wet: How much of the input (dry) vs. output (wet) signal to use for final audio output
  2. Latent Dimensions (#1-8)
    • Bias: Which direction (and how much) to bias a given latent dimension in
    • Scale: How much to weigh a given latent dimension
  3. Buffer Settings
    • Latency Mode: Size of buffer (in samples) to use for model inference -- higher values result in higher audio quality but more latency
    • Dry/Wet Latency Compensation: How long (in samples) to delay the dry signal by to compensate for latency
  4. Model Loading
    • Model File Path: Path to the RAVE model we wish to use

All the parameters in the Model Performance and Latent Dimensions categories can be bound to RTPCs.

Setup

(TODO: Set this up in CMake or something similar)

  1. Download and add library dependencies:

    1. Download LibTorch 1.11.0 from https://pytorch.org/ and place the libtorch directory in Libraries/torch (so that the full path to the library is Libraries/torch/libtorch)
    2. Similarly download and add ONNX 1.14.1 to Libraries/onnx (so the full path to the library is Libraries/onnx/onnxruntime)
    3. Download BS::thread_pool 3.5.0 and add the BS_thread_pool.hpp header to Libraries/bs_thread_pool (so that the full path to the header is Libraries/bs_thread_pool/BS_thread_pool.hpp)

  2. Run Wwise premake step. Below is a Windows example (you can substitute 2022.1.5.8242 for your version of Wwise):

python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" premake Authoring
python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" premake Windows_vc160

  1. Apply settings to the generated solutions (following the above example, the solutions would be RaveWwise_Authoring_Windows_vc160.sln and RaveWwise_Windows_vc160.sln): (TODO: Add these to PremakePlugin.lua)

    1. Apply general settings to the entire solution:
      1. Retarget solution > Retarget both RaveWwise and RaveWwiseFX projects to latest Windows SDK version (e.g. 10.0)
    2. Apply project-specific settings to the RaveWwiseFX project:
      1. Properties > General > C++ Language Standard: /std:c++17 (C++ 17 standard)
        • This is required for BS::thread_pool
      2. (optional, for debugging) Properties > C/C++ > Optimization > Disabled (/Od)
      3. C/C++ > All Options > Additional Options: /utf-8 /GR %(AdditionalOptions)
        • /GR enables run-time type information (used for std::dynamic_pointer_cast() in torch/nn/cloneable.h)

  2. Build for desired platforms, either using Visual Studio or wp.py directly. Below is an example for authoring and Windows (you can substitute 2022.1.5.8242 for your version of Wwise):

python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" build Authoring -c Release -x x64 -t vc160
python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" build Windows_vc160 -c Release -x x64
  1. Copy library files for authoring, substituting 2022.1.5.8242 with your version of Wwise:
python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" package Authoring --version=2022.1.5.8242 --additional-artifacts-file=additional_artifacts.json -copy-artifacts
  1. Package for desired target platforms. Below is a Windows example (you can substitute 2022.1.5.8242 for your version of Wwise):
# Package
python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" package Common --version=2022.1.5.8242
python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" package Authoring --version=2022.1.5.8242 --additional-artifacts-file=additional_artifacts.json
python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" package Windows_vc160 --version=2022.1.5.8242 --additional-artifacts-file=additional_artifacts.json

# Bundle
python "C:\Program Files (x86)\Audiokinetic\Wwise 2022.1.5.8242\Scripts\Build\Plugins\wp.py" generate-bundle --version=2022.1.5.8242
  1. Install from the Audiokinetic Launcher: https://www.audiokinetic.com/library/wwise_launcher/?source=InstallGuide&id=plugins

Licensing

This work is licensed under CC BY-NC 4.0, except for Wwise plugin scaffolding portions, which are licensed under Apache 2.0.