NVIDIA Webcam Tracker

In addition to tracking using OpenSeeFace, VTube Studio now supports high-quality, high-performance webcam tracking using the new NVIDIA Broadcast Expression Tracker for RTX GPUs.

Download for free here: https://store.steampowered.com/app/2178540/VTube_Studio__NVIDIA_Broadcast_Tracker

If you do not have an RTX GPU, you might want to check out the Google Mediapipe Tracker instead for comparable tracking quality.

(art by @GiruteaCafe)

This tracker is using GPU-accelerated NVIDIA machine learning technology, which makes it really fast and performant. Even at 60 FPS, the tracker uses very few CPU/GPU resources, so it's able to run smoothly along any other game.

How do I get the tracker?

The tracker is only available for Windows and only works with NVIDIA RTX GPUs.

To use it, please download the free NVIDIA Broadcast Tracker DLC from here: https://store.steampowered.com/app/2178540/VTube_Studio__NVIDIA_Broadcast_Tracker

Please keep in mind that the tracker files are relatively big (~2 GB), so make sure you have enough space on your hard drive.

Make sure the DLC is DOWNLOADED and INSTALLED before starting VTube Studio (checkmark set in the DLC window):

How to use this?

The NVIDIA Broadcast tracker can be started just like the regular OpenSeeFace webcam tracker. Just select "RTX - NVIDIA Broadcast" after clicking the "Tracking Type" button (see 1). This will only be available if you actually have a RTX-supported GPU.

1. Click this button to select the new NVIDIA Broadcast tracker. Will not be available if your GPU is not supported.
2. The tracking type. Only "Face Tracking" will be available here if you use the NVIDIA Broadcast tracker. Hand tracking is currently not supported when using the NVIDIA Broadcast tracker.
3. If you want to limit the tracking FPS, do it using this slider. Since VTube Studio interpolates the tracking data to 60 FPS, running the tracking above 30 FPS does not add much in terms of model movement quality.
4. Click this once after the tracking has initialized. Click while looking at the camera to set a neutral face pose and neutral blendshapes.
5. Click this to turn the external tracking preview window with the creepy face on or off (see 6).
6. The external tracking preview window. Can be useful for checking if the blendshapes are properly calibrated. Rendering the 3D face takes some CPU/GPU resources, so you should have it off most of the time if you don't need it. You can also close this window with the normal "X button" to turn it off.

Once the tracker has initialized and tracking has started, you can click the gray gear icon in the tracking preview screen (bottom right) to show the Windows webcam config window.

FAQ

Is this free?

It is free (with the watermark) just like the regular webcam tracking. No additional purchase needed.

Why am I not getting my full webcam FPS?

VTube Studio will interpolate tracking data to 60 FPS (or whatever VTS is running at), so even low-FPS webcam tracking data will look smooth. There is very little visible difference between your webcam running at 30 FPS vs. 60 FPS, so if you're not getting the full framerate, it shouldn't be a big issue.

However, technically the tracker should be able to run at the full FPS supported by your webcam (up to 60 FPS at least). If that is not the case for you, you can try the following:

1) Install the newest NVIDIA GPU driver

Make sure your GPU driver is up to date. You can do that via GeForce Experience or directly download the newest driver for your GPU from here: https://www.nvidia.com/download/index.aspx

2) Make sure "Low-Light Compensation" is turned off

If your camera has a "low-light compensation" mode, make sure it is turned off. Otherwise your camera may reduce the framerate get better lighting. This could of course decrease the webcam video quality, so be careful. For more info, check this video: https://www.youtube.com/watch?v=g02DBKjFXzs

You can open the camera settings window by clicking the small gear icon in the bottom right of the tracking preview window in VTube Studio.

This will open the following window: (make sure low-light compensation is off)

3) Make sure "Anti-Flicker" is set to 60 Hz

If "Anti-Flicker" is set to 50 Hz, your camera framerate will be capped at 50 FPS.

4) Install newest camera driver

If your camera has a special driver or software, download and install it. For Logitech cameras for example, that would be Logitech G HUB.

5) Make sure your camera actually supports that FPS

Double-check in VTube Studio that your camera actually has that FPS setting available. If it's not in the listed settings, your camera doesn't support it.

6) Make sure you don't have the FPS limit slider set to something low in VTube Studio

You can further limit FPS with the "Camera FPS Limit" slider on the webcam selection screen in VTube Studio. Make sure you have no low limit set there.

7) Try other video formats and resolutions.

If your camera supports multiple video formats, try the other ones. MJPEG for example can be a bit slow. You can also try lowering the resolution. There is very little to no tracking quality difference between 1080p and 720p.

8) Try a different USB port

Some older USB ports are not fast enough to support full 60 FPS streaming at high resolutions since that can be a lot of data. Make sure you're not using a USB extension cord or USB hub and the camera is directly connected to your PC's USB ports at the back.

9) Run VTube Studio as admin

Sometimes it can help to start VTube Studio as admin. Check this page for more info.

10) Check your camera in OBS

Open your webcam in OBS and check if it's actually giving you the expected FPS there.

Why does the tracker not start up even though my webcam light is turning on?

The tracker sends the tracking data to VTube Studio via your local network, so your Windows firewall may be blocking it. Make sure "ExpressionApp.exe" is added as firewall exception.

The file can be found in the MXTracker folder next to the Live2DModels folder. If it already is added, try re-adding it. Sometimes Windows invalidates these entries when the .exe file is modified by an update.

You can add it like this: (just an example, replace "vtube studio.exe" with "ExpressionApp.exe")

How does this compare to iPhone/iPad tracking?

In terms of tracking quality, it's very similar. Mouth-tracking is very accurate and so is eye-tracking. Blink-tracking works well. Wink-tracking is fine too, but as always, depending on your eye shape/size it may work better or worse. You'll just have to try it out and see for yourself.

The face rotation range of the NVIDIA tracker is also very big so it's unlikely to lose tracking in most situations, even with fast face movement. I tried shaking my head really fast until I got a headache and never managed to make it lose tracking.

It supports the same parameters you have with iOS tracking, including "Mouth X" and individual brow tracking, but it DOES NOT currently support cheek puff and tongue-tracking. The tracker is in active development, so it is likely that those will be added eventually.

So why would I still want an iPhone?

With the iPhone, all tracking is done entirely on the phone, so VTube Studio in your PC will run with minimal CPU/GPU usage. This new NVIDIA tracker is very performant, but there is still some CPU/GPU usage.

If you already have an iPhone and you're fine with the tracking and setup, I would recommend sticking with that.

Not having to use multiple devices is really nice though, so definitely give the new tracker a try as well, maybe it could be a good alternative for you.

Is VBridger supported?

For the initial release, probably not. But I'll definitely give the VBridger-team access to everything required to make VBridger work with the new tracker so they can add support after the initial release if they want. We'll have to figure out how that's going to work, but there are a couple good ways I think. Shouldn't be too hard.

I do not have any plans to add blendshape customization or curve editors directly in VTube Studio.

What CPU/GPU usage can I expect?

It depends on your CPU/GPU, but it should be fairly minimal, even at high framerates. On my RTX 3080, running the tracker at 60 FPS and at 1920x1080 webcam resolution, both CPU and GPU usage stay below 10%.

A lot of the tracking system (but not all of it) can be executed on the machine-learning-optimized tensor-cores of your NVIDIA RTX GPU. They aren't used by most video games, so they don't interfere with your gaming.

Will the NVIDIA Broadcast face tracker be improved?

The latest info I have from NVIDIA right now is that "virtual avatars and tracking" are a big priority for them right now, so it is likely that there will be improvements to this tracker (which is kind of a nice change seeing how Apple haven't ever updated their face tracking).

So improvements in quality/speed are likely.

What about hand tracking?

That isn't currently supported with this tracker but NVIDIA are experimenting with various tracking types, so I would not be surprised if it's added eventually. I don't have any info about it though.

What do I need?

The only requirement is a NVIDIA RTX Series GPU and a webcam (any webcam will do). All RTX GPUs are supported, so any RTX 20-series, 30-series or (upcoming) 40-series card will do, including laptop GPUs. GTX GPUs or GPUs from other vendors are not supported. Current Steam surveys show that about 30% of users have RTX cards and that number is increasing with each new NVIDIA GPU generation.

If you are not sure if you have an RTX GPU, please follow this guide to find out. If your GPU name starts with "NVIDIA GeForce RTX", you have a supported GPU.

Does this work with glasses?

I've tried it with glasses and the eye-tracking and wink-tracking seems to work fairly well. Of course, it depends on the webcam placement and lighting too.

Is the normal webcam tracking (OpenSeeFace) still available?

Of course! It is available and will always stay available as the main webcam tracker. This is just another option you can try out.

Can I use my existing models or do I need to change anything?

Parameter ranges/setups will be as close to iOS tracking as I can get them, so all your existing models should work without any changes (except cheek-puff and tongue-tracking since they are not available with this tracker for now).

Should I buy an RTX GPU for this?

Up to you. The current OpenSeeFace tracker is already really good, so if that works for you, there's no need to update your GPU for now (unless you were going to do it anyway).

I would recommend waiting until the update is out and then check out the comparisons people will make. I think this new tracker is definitely a nice option to have available.

Also, if you already use VTS iPhone tracking or VBridger and you're happy with it, there is probably not much of a reason to change your setup for now.