Network refactoring

[marchermans]

TLDR.

• Reworked networking, now completely based of of CustomPacketPayload and native to Minecraft
• Created events to register custom network payload types
• Implemented new negotiation mechanic for network payload types
• Fixed several bugs with the negotiation mechanic (aka there being for example a 0 in the minecraft:register payload type)
• Create an event to handle custom configuration tasks

Introduction

Reworked networking

Welcome to the reworked networking pull request for NeoForged.
The changes in this pull request implement the configuration network protocol designed by @modmuss50 with some minor modifications. I'm sharing more on those below.

Tip

The NeoForge @neoforged/maintainers did perform an analysis of the networking protocol. It is a bit rough, but the resulting packet flows, and analysis comments can be found here

Consequences of the split packet type introduced in 1.20.2

During the introduction of 1.20.2, Mojang created in practice two distinct packet systems, one set known as the login packets and one set known as the play packets. Within those play packets, a unique phase was created; during this phase, the client configuration is supposed to happen.

During the port to 1.20.2, we needed more time to address these unique structures within the packet layout that Mojang introduced, create APIs that allow modders to use these new packet systems to their fullest, and give access to this unique configuration phase. This PR achieves this.

SimpleChannel and EventChannel

Original Forge contained two different ways of implementing a custom network channel. A simple registration-based approach is known as SimpleChannel, and a system that fires events for each packet received is known as EventChannel.
To simplify using networking and make interacting with new packet layouts easier, it was decided to rework both implementations into a single system, combining the best of both worlds.

Implementation

New network payload handling

The system is based on the CustomPacketPayload definition that Mojang uses to represent the content of custom packets. Internally, they register their custom payloads, used mainly for debugging, to a map. We need to extend this map so that a modder can send and receive a custom implementation of this CustomPacketPayload. This is what a significant part of this PR achieves. Modders can introduce new CustomPacketPayload implementations by registering them during the RegisterPacketHandlerEvent.

The registrar

Any modder can request a registrar for any namespace he or she desires; however, it is recommended that a modder only request a registrar for his or her namespace. Once a registrar is retrieved from the event, the modder can then configure the registrar with two different options: versioned(String version) to configure a version for all payloads registered after the call and optional() to mark all payloads registered after that call as not requiring a receiving side. An example of the registrar configuration can be found here:

@SubscribeEvent
public static void register(final RegisterPacketHandlerEvent event) {
    final IPayloadRegistrar registrar = event.registrar(NeoForgeVersion.MOD_ID)
            .versioned(NeoForgeVersion.getSpec())
            .optional();
}

Note

The registrar is an immutable object; calling versioned(String version) or optional() on an instance of the registrar will cause a new instance with the desired configuration to be created.

Caution

The registrar loses its validity the moment the scope of the event has been left. Registering payload handlers outside of the event handling scope will result in those payloads not being known to the system and not being sent over the connection.

The registrar offers six different endpoints, three pairs of 2, to register new payloads. A pair exists for the play phase, one pair for the special configuration subphase, and one pair of methods for both.

Important

It is impossible to register custom payloads that should be sent during the login phase of the connection. And this PR offers, as such, no infrastructure to achieve this.

Within each pair of registration methods for a phase, two variants (as such, six methods) are available. One that registers the same handler for both sides of the connection and one that takes a consumer, allowing for the configuration of single-sided or differentially handled payloads.

An example of the signature of the methods for the configuration phase is as follows:

<T extends CustomPacketPayload> IPayloadRegistrar configuration(ResourceLocation id, FriendlyByteBuf.Reader<T> reader, IConfigurationPayloadHandler<T> handler);

<T extends CustomPacketPayload> IPayloadRegistrar configuration(ResourceLocation id, FriendlyByteBuf.Reader<T> reader, Consumer<IDirectionAwarePayloadHandlerBuilder<T, IConfigurationPayloadHandler<T>>> handler);

For the play phase, similar methods exist.
For payloads that are supposed to be sent during both the play phase as well as the configuration phase, there also exists a pair, however here, the handler is a common supertype of the handling callbacks, and the handler only has a reduced common superset of the information available to both phase handlers to it available.

Payload discrimination

When considering this system, you might, right-fully so, ask how the system keeps the different payload types apart from each other. This is achieved by the system it-self. It writes a discriminator ID to the connection before invoking the actualy writing mechanic for a payload. Conversely on the client side, the discrimininator is read first, before a reader is looked up so that the rest of the payload can be read.

Note

The system handles reading and writing the discriminator it-self, as a modder you do not need to take care of this yourself.

The value of the discriminator during writing is retrieved from the CustomPacketPayload#id() method, and is not allowed to be null.
The value against which the id, read from the connection, is compared to find a reader, is the one that is given to the registrar as the first argument.
It is as such of the upmost importance that you return the same value, as in the same resource location, from both the id() method on the payload instance, as well as given the same value to the registrar when registering your payload.

Tip

We recommond that you store your ID in a public static final ResourceLocation ID = new ResourceLocation("mod_id", "payload_id") field, and reference that in both places.

Warning

Given that the id is used as a discriminator, it is important that you use a unique value, especially for the ResourceLocations path section for each payload type. If you try to register the same id twice, the registrar will throw an exception. If you try to register an id with a namespace other then the one the registrar is for, the registrar will throw an exception. You are free to request registrars for other namespaces then your own.

Payload reading

Payload reading happens via the vanilla method, implementing the FriendlyByteBuf.Reader<T> functional interface. During the registration of the payload type, as can be seen above, you need to pass an implementation of this interface so that a new instance of the payload can be created by the system when a custom payload packet with it as payload arrives on the receiving end.

Tip

As we recommend that your payload implementations are records in java, instead of classes, we also recommend that you create a custom constructor with the record, to read the records fields from the buffer. This constructor can then be passed as a method reference for the reader implementation. So if SimplePayload(String somethign) {} is your normal record, then adding SimplePayload(FriendlyByteBuf buf) { this(buf.readUtf()); } as a constructor to the SimplePayload record will allow you to pass it as a method reference SimplePayload::new to the registar when it asks for an implmentation of FriendlyByteBuf.Reader<SimplePayload>

Warning

There is no guarantee with respect to which thread the reading callback is invoked upon. It is as such important to note that the method can be called on many threads in parallel if the same packet is received on many connections simultaniously.

Payload writing

The CustomPacketPayload interface contains a method: write(FriendlyByteBuf) method. This method is invoked when it is time to write your payload to the network connection. There is no guarantee on the thread this is invoked from.

Warning

There is no guarantee concerning which thread the writing method is invoked upon. It is as such important to note that the method can be called on many threads in parallel if it is being send over several connections at once.

Caution

Payloads are only read and written if sent over a connection. This means that the host of a single-player world (even if exposed to LAN) has packets and, as such, payloads transferred in memory. This means that for those payloads, no write method is invoked, and no reader is called. Only the handler is invoked!

Payload handling

Once a payload has been written, transmitted and read, the payload handler is invoked. This handler is again looked up with the id of the payload, and then invoked with the context of the receiving end. In practice each payload handler takes two arguments:

• The payload instance
• The context

Warning

Processing of the payload, as in the invocation of the handler callback, happens on the network thread, and can as such happen in parallel with other payloads of the same type being handled. If you need to ensure that the payload is processed on the main thread, serially, see the ISynchronizedWorkHandler available in the context under the workHandler() method.

The context

The context contains information, callbacks and entry points, to access the surrounding network system, the main thread, as well as handling processing other packets, or completing configuration tasks.

ReplyHandler

The reply handler can be used to quickly send a payload back to the sender. It is for example usefull to send and answer to a query packet, or simply send an acknowledge that you received and processed the payload. You still need to register the return payload.

PacketHandler

In case you implement a packet splitting mechanism, whether that splits on full vanilla packets, or custom packet payloads, the IPacketHandler interface gives you access the the start of the processing pipeline, allowing you to process other payloads immediatly.

Note

This does not transmit payloads, it purely allows for the receiving end to process additional packets or payloads constructed in memory during the processing of your payload

The packet handler also gives you access to a disconnect(Component) method allowing you to terminate the connection at will, showing the given component as reason to the user.

WorkHandler

The work handler allows you to schedule work on the main thread of the receiving side. This might be the Minecraft class instance if the logical receiving side is the client, or the MinecraftServer instance if the receiving side is the server.

The system uses CompleteableFuture instances so you can schedule different follow up tasks, if need be.

PacketFlow

The context will indicate via the packet flow, what the receiving side currently is. If it is a server bound flow, then handler is currently being invoked in the context of the server. Is it a client bound flow, then the handler is currently being invoked in the context of the client.

ConnectionProtocol

The current active connection protocol. Usefull if you have raw bytes of a packet wrapped in your payload, allowing your handler to decode the inner packet or payload, before passing it to the IPacketHandler for processing.

ChannelHandlerContext

The netty channel handling context that is currently processing the network connection on which the payload was received.
This context can be used to retrieve the raw underlying connection via ConnectionUtils or again be used to process raw bytes of inner packets and payloads.

Sender

This is an Optional containing a player, if and only if the handler is being invoked on the server side, but not during the configuration phase (there is simply no player instance yet, as such there is no way to get the information)

TaskCompletedHandler

This is a special contextual value only available to payloads for the configuration, which are specifically registered as such, to indicate that a specific configuration task has been completed, and that the next one can be started.

Future additions to the contexts

We are fully aware that these entry points might not be all information you as a modder need to process a packet. In general it is pretty easy to extend the interfaces and records. They were specifically designed to allow for simple PRs in the future to add to them, so please do not hesitate to create a quick PR to add your needed data to the context.

Packet Sending

In tandem with the refactoring of the channel registration mechanic we added new tools and systems to allow you to easier send a custom payload to different targets.

PacketDistributor

This wrapper class now has the ability to process custom payloads solely. Its instances and targets can be passed around since they are immutable. Several methods on extension classes will accept these, to facilitate easy transfer of payloads.

Extension objects

We extended several vanilla types allowing for them to accept payloads as well, not just packets. Examples are chunk sections, listeners, entities and players.

Netty information

In contrast with the past, we now store a lot of information related to the connection, for example the negotiated payload types, on the connection object itself. To this accord we added several attributes to store this information in. Allthough the attributes are considered part of the internal API, if somebody wants to dig through the internal guts, they should feel free to do so, however there is no guarantee that we would change them around in future releases, even outside the BC window.

Configuration tasks on client join

Tasks

Vanilla now provides a centralized way to perform tasks and jobs that need to be performed when a player joins. The player won't be instantiated or added to a world untill these tasks are completed.

Under normal circumstances these tasks are implementations of the ConfigurationTask interface, with a single method: start(Consumer<Packet<?>>). However, this is subpar in our situation. Modders should never really have to touch raw Packets to perform their duties, only payloads. And as such it was decided to have modders implement the ICustomConfigurationTask interface from neoforge itself. This provides a wrapper around the ConfigurationTask signature and allows for sending payloads instead of packets, by implementing run(Consumer<CustomPacketPayload>) instead. The given consumer will then automatically convert the payload to a packet and send it to the client that is being configured.

Note

In practice an instance of ICustomConfigurationTask is also an instance of ConfigurationTask as one extends the other. But to provide the ability for this to be a functional interface, the method Start is implemented as a default implementation. It is not recommended to also override that.

OnGameConfigurationEvent

This event is fired to collect all tasks that should be run, and allows for the registration of ICustomConfigurationTask instances to the listener. It is not possible to register the vanilla ConfigurationTask instances.

Tip

This event is fired on the mod bus, to preserve dependency order. Given that configuration tasks can only be ran in order of registration, you can safely assume that configuration tasks of your dependencies have been ran before yours.

Forge changes:

Moved configuration sync, registry sync, and tier registry sync to configuration phase tasks.

Bundle packet processing

In 1.20.2 Mojang introduced the bundling system for packets, which is a core component that allows for packets to be processed together. And started using sparsely. We anticipate modders want to use this system, so we adapted it to accept custom payloads. You will find a sendBundled(CustomPacketPayload[] payloads) method on the ServerGamePacketListener

Warning

Packet bundling is only supported during the play phase of the network protocol. It can not be used during the configuration phase of the protocol.

Opening menus with custom data

In the past Forge supported opening UIs from the server side with additional data, via NetworkHooks.openScreen(...). this system has been moved and is now part of the server ServerPlayer extension. You can call the method openMenu with the same parameters.

Spawning entities with custom data

The previous networking implementation allowed for the spawning of custom entities via an overridden method on the Entity class: getAddEntityPacket. Modders that wanted to support custom additional data to be processed on the client side when the entity spawned, could override this method and return a packet from the method: NetworkHooks.getEntitySpawningPacket(...)

This system has now been refactored (as Mojang spawns and processes the entity packets with a bundle). The core of this new framework is the method sendPairingData(ServerPlayer, Consumer<CustomPacketPayload>) on the Entity class. There are now two methods to configure this.

Using a custom payload

By overriding the method and invoking the consumer with a custom payload, you are guaranteed that your payload will be processed immediately after the spawning packet. You are free to do whatever you want, however, we recommend you at least transfer the entity id, as vanilla does, to retrieve the entity instance when your packet arrives.

Using the IEntityWithComplexSpawn interface

Implementing this interface on your entity forces you to implement two methods: writeSpawnData and readSpawnData. These methods, respectively, are invoked when an entity spawn bundle is generated and when the entity has been spawned.

Removal of custom entity creation code in EntityType

It is no longer possible to use the entity spawn packet code mentioned above to create a different entity class on the client side. This was due to the refactoring of the payload mechanics. And the reliance on the vanilla spawning bundle.

Forge Packetsplitter

This PR reimplements the vanilla packet splitter forge, previously contained in Forge, and expands its usage to any packet (other than the packet sent when stuff is split).

Due to the new mechanics, a packet or payload is now roughly split into 8MB chunks, as per the vanilla protocol limitations.

Modlist transfer

Currently, the new protocol does not support sending the mod list across and only works on a channel and registry content basis. However, we do intend to work with the Fabric team on expanding the protocol in such a way that the server is made aware of what kind of mods the client has installed.
We deem this necessary to allow server owners to block mods that would allow cheating, for example, an XRay mod, etc.
This will be added in another PR after further reviewing our stance and implementation possibilities.

---

Todo:

• Implement registry sync failure disconnecting and messaging
• Discuss futher contextual components to the ServerConfigurationPacketListener, as this is the only piece of information given to the OnGameConfigurationEvent. We already exposed on that listener type if the connection is vanilla or not. Maybe the players UUID should be exposed as well? Please discuss this in the comments.
• Test further scenarios where mismatches can occur
• Is modlist mismatch testing even needed? Should we not more think about this in the form of content, aka network payload types, registries and their contents, tier sorting etc?
• Improve support for bundle packets
• Introduce endpoints to detect the payload state

---

Follow up PRs

• Introduce a debugging tool
• Expand configuration protocol to communicate the client mods

[robotgryphon]

How are these RegisterPacketHandler events sorted? A concern is that if any listener can request any registrar, it becomes trivial to stomp on changes from other mods accidentally, ESPECIALLY if a mod wants to "add-on" to another mod's existing registrar, given the whole system is "immutable" --- can this be elaborated on?