Subframe rollback

[JulienBernard3383279]

This is meant as a permanent discussion hub for the subframe rollback fork and how it relates to Slippi / hypothetical eventual integration. This isn't currently ready for merge and it's unclear if it ever will considering how the feature works.

Explanation

Say player A and player B take 2.4 frame lengths (2.4 * 1/60s = 80ms) to transmit inputs to one another and they are in sync timing-wise. They play on delay 2. At frame n, they will send the information for frame n+2. This information will arrive at t = n+2.4. This means that on frame n+2, both players will use the pad data of frame n+1 (sent on t = n-1), before fixing it on frame n+3 by rolling it back to use the definitive pad data for n+2 instead (acquired on t = n+2.4).

This is a waste, because on t = n+2, we could be using the pad data for t = n+1.6 instead - if only we had it. It's not a definitive input, but it doesn't matter: it will get rolled back anyway. That input is "closer" to the input for t = n+2, however you'd define a distance between gamecube controller pad states. 60% of the analog movement done between t = n+1 and t = n+2 is done - meaning a 60% chance to know about an animation change that happened on that frame because of such a movement. The same applies to digital inputs: if an A press occur between these frames, there's a 60% chance it occurs in the first 60% of that time window.

Of course we don't want to send anything that we learn about the pad state as soon as we learn of it as that could mean sending up to 1000 messages per frame (the currently fastest polling setup for Melee reports to Slippi at 1000Hz). Instead, we'll send only pad states with meaningful updates relatively to the latest known state. Every time an adapter libusb transfer completes, the contained input is judged for whether it's worth sending. This happens in KristalInputJudge. Everything related to this dev that needs to be told apart from a Slippi counterpart is named Kristal. People familiar will my earlier devs and electronics will get it !

Inputs judged worthy of being sent as of this first version are:

• A B and Start presses
• X Y Z L R presses and releases
• transitions from no shield to analog shield
• getting out of a X or Y deadzone with control stick
• center to >= dash thresholds
• center to >= tap jump threshold
• center to >= crouch threshold
• getting out of a X or Y deadzone with C stick (aerials &co)
• center to >= C stick smash thresholds

This is a bit more lenient than IPM counting, which for top level players is about 10 inputs per second. I doubt it ever goes past 20 / second, and I expect it'll be much lower on average, which means that compared to the usual 120 Slippi network messages per second, it's a 10~15% overhead.

Architectural hacks

These network communications are **driven by the polling thread. A callback is injected in GCAdapter by SlippiNetplayClient.

This highly questionnable architecture isn't without consequence and requires a number of hacks to function. It's almost like USB comms entities and netcode entites aren't supposed to communicate directly.

The GCAdapter threads have no ideas whether Netplay is running. A callback is injected by SlippiNetplayClient that's called with the data of worthy pads.

The GCAdapter has no idea what port is being used by the game - if any, as it is several layers away from this information which depends on which controller is used to open the CSS.
The last controller to have pressed start is considered to be the used controller. For the record, Start is necessary to press to enter the Direct code.

The GCAdapter doesn't know about the origins value, and since the pad data is extracted from the Slippi ASM, it is already representend in internal pad format and correct by origins value. Because of this it's necessary to attempt to mimick what the pad representation translation function used internally for this port, along with origin adjustments. The translation function is convertToInGamePadData in SlippiNetplay.cpp.
The origins are more tricky. At the moment the origin query in the SI tells the adapter it happened, which looks at the latest known pad state and uses that as origin - I think it should be possible to get the correct pad used as origin. Right now the origin might be +-1.

Known "bug" ?

Currently, when passing the inputs for frame n, the latest (highest version, then highest subframe) Kristal input within [latestSlippiInput, n] is passed as prediction. This prediction is used for all rollback predictions. This means that if at frame n we somehow don't have the inputs for n-1 but have a Kristal input for subframe n-0.8 (which is improbable), we're going to predict frame n-1 with the the pad data for n-0.8. It's weird, but I'm pretty fine with that. I think it's better to use data that's 0.2 early than 1 late.

Notes on WinUSB vs HID

The evaluation of inputs' subframe (i.e a USB transfer completes - how to evaluate "now" is "frame 26.78" ?) build on the timing reduction dispersion devs of #211. That dev, and this one, only work with WinUSB, as it lets you handle completed transfers, thereby providing timing data, whereas HID hides it from you and only support being asked what the latest known report is. This means this dev only work with GCC+adapters (or whatever pretends to be an adapter)
It would be possible to poll HID controllers (worth it here, as opposed to the timing dispersion stuff), port the timing dispersion framework for binding engine polls with time points, but it clearly won't be easy and I won't do it for v1.

This also means the Reduce Timing Dispersion mechanisms need to be active. I've forced them to true and disabled the checkbox (clearer than hiding it imo)

Gains

IWith the time unit being the frame length, the information travel time -> average amounts of rollback are as follows:

Traditional rollback:

• t in [0, delay] => 0
• t in ]delay+n, delay+n+1] => n+1

Subframe rollback, assuming all the useful information is sent:

• t in [0, delay] => 0
• t > delay => t - delay

In practice, there are no jumps in statistical gain: players will be ahead of one another and may switch over time. There isn't even an initial synchronization mechanism to assume they are closer to phased within a frame to begin with.
In Slippi currently, every 30 frames, if one player is ahead of the other by more than 0.6 frame, it stalls for one frame (side note: the timing dispersion framework is made aware of with, adjusts the time point - frame matching and sends versioned messages within a frame for this case).

Assuming players are ahead of each other in a uniform distribution within [-0.5f, 0.5f] is a decent approximation (although 0.5 to 0.6f is a stable range and it's possible to get away from this range in the 30 frames before the next sync).

With this model, average amounts of rollback are as follows:

Traditional rollback:

• t in [0, delay-0.5] => 0
• t > delay-0.5 => t - (delay-0.5)

Subframe rollback:

• t in [0, delay-0.5] => 0
• t in [delay-0.5, delay+0.5] => (t-delay)^2 /8 + (t-delay)/2 + 1/8
• t > delay+0.5 => t - delay

The takeways being:

• this only start being useful (in this model) at t = delay-0.5, which is 50ms for delay 2.
• at t = delay, usually 67ms, this reduces the average rollback length to 1/8 from 1/2. (It's not linear: if the player is late, he doesn't rollback, if he's early, he rollbacks by between 0 and 0.5 frame lengths uniformly)
• past delay+0.5 - usually 83ms - this reduces the average rollback length by 0.5.

Additionally, having the network messages triggered from the polling thread means any artificial delay induced between the polling thread and the engine is bypassed by this mechanism. Such delay is induced by the timing dispersion devs, that induce about 0.1 frame length of delay to make room for enforcing better pacing of the inputs used by the engine. This means the subframe rollbacks input are sent before they're even exposed to the engine, on average by the length of the input stabilizer delay, in our case 0.1f.

This doesn't mean 0.1f of input delay is not experienced anymore, it means it's now acts as part of the netcode buffer. So while before, turning on input stabilizer made the situation the same from a netcode perspective with an extra 1.6ms of input delay locally, now it still adds the delay to your local input lag, but the netcode buffer is 1.6ms larger. "That" part of the buffer isn't as good as the rest though, as it only acts as a buffer for the inputs judged worthy of sending in the prediction stream. Note that it shouldn't cause more rollbacks though, since a correct prediction won't trigger a rollback.

So, if you were using timing dispersion reduction, the maximum ping reduction equivalent to the average rollback length reduction is 20ms, reached at ping 86.67ms. Otherwise it's 16.67ms, reached at ping 83.33ms. (again, that's assuming the model is correct - in reality the ping to reach the maximum reduction is later, and the minimum earlier)

Security concerns

This method of operation brings huge cheating concerns due to giving power to the client. In traditional rollback, if you send a wrong information, you desync. Here you're expected to be "nice", and send completely optional information to improve your opponent's experience. You could not send it, or worse, you could send bullshit. https://gfycat.com/raretintedarthropods

This is a strong case against ever using this in competitive settings and keeping this purely for training between trusted parties. A serious log analysis mechanism would have to be developed for it to be considered viable for use in matches with any stakes, and even then, it would never be perfect.

Integration

I'm currently considering releasing this as a custom build, in which case I'll make it as clear as possible the Slippi team doesn't offer support for it, although I'm undecided on how to proceed.
If I do I may try to make a short explanation video which will take me some time (as nobody but Slippi people properly understood text explanations).
I only intend for this to be used for Direct training - Teams should work but I never tested them.

I've disabled access to Unranked in this build. I'm also considering adding some sort of handshake at the beginning of any direct play between all players to confirm they are all on this custom build before they're allowed into the game. I don't want anyone (including me) to have to deal with bugs arising from games between this build and the official one, even accidentally.

I hardly see why there would be bugs so long as this is based on the Slippi head though. There would just a flurry of messages with an unknown ID received by the vanilla build.

Tests

Aside from my personal tests, this was tested by 3 player groups. One was extremely positive about the improvement, one didn't say much, and one was overwhelmingly positive, but the latter 2 groups' reports' validity is questionnable as their ping changed respectively by +10ms and -40ms between vanilla and this build (which makes no sense to me whatsoever). Unfortunately neither switched back and forth then so I couldn't confirm whether this was tied to the builds and not uh... magic rocks.