What kind of networking should X game use?

[erlend-sh]

Once again I've found myself linking to this excellent networking explainer when someone making a multiplayer game showed up with questions about best practice. Having it stored on an unindexable Discord server within a thread does it a disservice, so I'm gonna repost it here in hopes that it can soon make its way into a documentation page.

From Joy @maniwani:

What kind of networking should X game use?

IMO answering these three questions can decide for you.

1. Deterministic or authoritative?

Think about:

• How complex is the simulation? How many entities need to be replicated?
• Is there hidden information? Is it important that nobody can access it before they're supposed to?
• Do any of the libraries you want to use introduce non-determinism?

With deterministic replication, clients exchange inputs (directly or indirectly) and independently simulate the world.

With authoritative replication, each element of the game state has an owner who is its source of updates. Usually, the server owns everything ("server auth") and clients only send inputs to it, but ownership can be distributed among clients ("client auth"), and then they can send state as well.

Let's call 'em DR and AR.

If you're making an RTS game, consider DR. Players might control hundreds of units at once. AR would need lots of packets to exchange information about all of them, but bandwidth is often limited. DR only has to exchange inputs, no matter the number of entities, so DR is generally better for games with lots of networked entities. (There are RTS games that use AR though.)

Some exceptions to that heuristic would be games with a massive number of players (in the same session) and games with massive, open-world environments. With DR, each device generally has to simulate the entire world. I don't think anyone can run all of Teyvat at 60 FPS. AR more easily allows for cherry-picking what to send to and simulate on each client, making it the better choice in those scenarios.

For the same reasons, AR is generally better for games that require hidden information. If you're making an online poker game, it'd be really bad if you could use a program to leak another player's hand before they reveal it. DR is probably the wrong choice here, since every player will have the entire game state in memory. That said, DR could work if you are careful to send hidden information as input only, and only when it's ready to be revealed.

Determinism is hard to guarantee

In practice, before you compare DR and AR, you should probably ask yourself if DR is even a feasible option.

Dependencies

To be deterministic, the functions you use in your simulation should always return the same results given the same arguments. Ideally, those functions are also pure/stateless, i.e. all of their arguments are explict. Being stateful is OK if you can access, modify, and clone that state.

Unfortunately, oftentimes you can't.

Many physics libraries, for example, have internal data structures to speed things up (alongside the colliders, bodies, and joints supplied by the user), like BVHs for the collision detection and islands for the constrained dynamics. PhysX, Bullet, and Box2D are some of the most popular libraries, yet none of them (or at least the game engines that integrate them) seem to expose their internal state, which makes them incompatible with DR when you also want prediction since you can't rollback properly. (This is inconvenient for AR as well.)

Math

Floating-point math is a notorious source of non-determinism.

• Floating-point math operations round their results (because precision is finite), so floating-point math is non-associative. (a + b) + c and a + (b + c) are not equivalent.
• IEEE-754 exactly defines the binary representation of floating-point numbers and how addition, subtraction, multiplication, and division should be calculated, so they (should) match on all platforms. However, it does not constrain a bunch of other operations (e.g. sqrt, sin, cos, tan, exp, log, etc.), so hardware implementations of those operations can differ.
• Compilers basically don't comply at all with IEEE-754 on NaN values.

But good news! It is possible (apparently) to address those issues. If you can:

• Make sure the compiler can't mess with your floating-point expressions.
• Make sure the rounding mode is consistent.
• Mostly stick to add, subtract, multiply, and divide.
• Source any transcendental functions (e.g. sqrt, sin, log, etc.) from a library like libm (piecewise polynomial approximations) instead of std (hardware intrinsics) to get consistent results.
• Never rely on the exact bits in a NaN value.

Then you should get results that are consistent cross-platform.

2. To predict or not? And how much?

Think about:

• Does the game run in real-time?
• Is visual weirdness from mispredictions OK?
• Is visual weirdness from lag compensation OK?
• What can clients afford to simulate?

Nobody likes lag, so if your game runs in real-time, your game clients should try to give players input feedback as quickly as possible. Prediction (rollback) is the most common way to hide the network round-trip time. Not the only way though.

On one hand, fighting game players love rollback and want it in every game. On the other hand, mispredictions can look glitchy. RTS games often run in lockstep (no prediction) but add artificial input delay to give all players just enough time to exchange inputs without stalling (hence "delay-based netcode"), then they try to hide that delay with animations.

DR clients mostly predict everything or predict nothing. AR clients more commonly predict some things and not others, with the server doing "lag compensation" to resolve interactions between predicted and non-predicted entities. Very common in FPS games but causes those "How did that hit connect?!" moments.

Predicting everything really helps in fighting games and physics-heavy games like Rocket League because players can react to what's happening and interact with objects (i.e. slam into a giant, flying soccer ball) at the same time without any weirdness, but it's more expensive.

3. Client-server or peer-to-peer?

Think about:

• Just how vague networking terminology is.

If you're asking in the sense of "star network or mesh network?", you really can't go wrong with the star network topology. Mesh networks:

• Consume more bandwidth (more connections, more data). Some consumer-grade internet plans may not have enough.
• Encounter more NAT/firewall issues (more connections, more problems).
• Tend to make replication more complicated by introducing edge cases that inspire complex consensus models to resolve.

If you're asking in the sense of "dedicated server or client host?", that's more of a money question. The server is just a role. Whether that role is performed by a player's machine or a datacenter rack shouldn't affect any code. You would need something to do NAT traversal (e.g. ICE, STUN, TURN) to support clients hosting (until IPv6 becomes the dominant protocol), but other than that, it's all the same.

Also: Netcode Terminology

[lokimckay]

Super clear, fantastic resource thanks for posting.

RE mesh network bandwidth: "Some consumer-grade internet plans may not have enough"
Does this also apply to DR (only inputs sent)?

I suppose it would depend on the frequency and volume of inputs, but I'm not sure what a good rule of thumb would be to consider transitioning from mesh -> star.

[Plecra]

DR clients mostly predict everything or predict nothing

It's good to keep in mind that you do still want to design game mechanics for latency resolution. 8 Frames in 16ms: Rollback Networking in Mortal Kombat and Injustice 2 is directly covering a deterministic rollback system, and I Shot You First: Networking the Gameplay of Halo: Reach also directly discusses mechanical design for latency compesnsation. Id recommend both, along with Overwatch Gameplay Architecture and Netcode.

TLDW, mechanical interactions in a game will often carry quite drastically different expectations from a player, which lead to different kinds of latency resolution feeling "right": The simplest example is that you want damage to be immediately predicted, to understand whether you're connecting right, but you never want an enemy to be predicted to die and then be resurrected by rollback. Instead you want to tune the simulation to make sure that deaths always happen only based on input that has been confirmed. The Mortal Kombat talk covers this in the case of special moves too. The halo talk has an example about activating shields, where they decided to have the ability activation have a builtin delay within the game, where the character is animated to initiate the shield, to make sure that attacks against a shield didnt have to be mispredicted. You want to be applying all of these design decisions with DR too.

[captkirk88]

Great topic!

A long time ago I delved deeply into trying to come up with the perfect networking strategy. There is no perfect networking strategy. All have trade offs. Unfortunately I lost that source code and PC (this was before the cloud and Github wasn't popular yet, remember SVN? I won't go there.) My approach was that the Server kept entity state of everything based on player environment (this requires a lot of RAM on the server unfortunately, which could be mitigated with databases).

Server acted as a simple relay of inputs after player environment state was sent. This is possible using a "wait to spawn" delay but you run into the trouble of desynced entities. Easily resolved by playing "catch-up" by sending the player inputs that happened while waiting until you catch up and sync. Don't sync to the most up-to-date state because then you will hit the frustration of never catching up. This approach requires you to design around the limitation of all entity handling is done by input. A player interacts with a NPC, they had to send input (mouse click) to do that. A player walks over a trigger on the map, enemies spawn. Benefit is the client side knows what will happen with the NPC and what that trigger will do. Server is only a relay. Downside is client has way too much control.

After a lot of research and destroyed relationships, I found there is no best approach. The only approach is what works with the kind of game you are designing and the resources you have available.