1.21.5 - Add support for new protocol checksum & chatIndex #1386
Conversation
| // Override and extend logSentMessageFromPeer to increment index | ||
| const originalLogSent = client.logSentMessageFromPeer | ||
| client.logSentMessageFromPeer = function (chatPacket) { | ||
| // Include chat index in outgoing player_chat packets | ||
| if (client.supportFeature('chatIndex')) { | ||
| chatPacket.index = client._chatMessageIndex++ | ||
| } | ||
|
|
||
| return originalLogSent ? originalLogSent.call(this, chatPacket) : true | ||
| } |
There was a problem hiding this comment.
@extremeheat just to clarify, why is that change not good?
| const calculatedChecksum = calculateChecksum(packet.previousMessages) | ||
| if (packet.checksum !== 0 && packet.checksum !== calculatedChecksum) { | ||
| debug('Chat checksum mismatch', packet.checksum, calculatedChecksum) | ||
| raise('multiplayer.disconnect.chat_validation_failed') | ||
| return |
There was a problem hiding this comment.
I'm not sure we should be enforcing all these checks at the library level as opposed to the user level. We did add some abstractions to chat for the purpose of not having to reimplement complex logic on the user's side. Perhaps it would make sense to have these kind of kick checks behind a flag.
There was a problem hiding this comment.
@extremeheat is there an existing flag that these kinds of kick checks could be behind or would a new flag have to be created?
| client.emit('error', new Error(`Chat index mismatch. Expected: ${client._expectedChatIndex}, got: ${packet.index}`)) | ||
| client.end('Chat message received out of order') | ||
| return |
| // Add checksum for 1.21.5+ | ||
| if (mcData.supportFeature('chatChecksum')) { | ||
| params.checksum = options.checksum !== undefined ? options.checksum : calculateChecksum(client._lastSeenMessages) |
There was a problem hiding this comment.
nit: if both the if branches rely on same code you can avoid the duplicate code here by putting it up top
| // Calculate checksum for the lastSeen message signatures (1.21.5+) | ||
| const calculateChecksum = (lastSeen) => { | ||
| if (!mcData.supportFeature('chatChecksum') || !lastSeen || lastSeen.length === 0) { | ||
| return 0 // Default value for backward compatibility |
There was a problem hiding this comment.
Can you elaborate here what you mean by "backward compatibility"?
|
I still have an issue on the client side. I'm investigating, it seems related to the login packet but I checked the protocol (directly by decompiling the client and comparing with 1.21.4) and nothing as changed (at least from my POV). |
|
part of 1.21.5 support; tracked at PrismarineJS/mineflayer#3641 |
@RomainNeup Is it alright if you could provide a quick overview / steps on how to reproduce this issue? I would be willing to look into it further. |
extremeheat
changed the title
There was a problem hiding this comment.
The big changes for nmp seem related to chat.
There's a new checksum and a new globalIndex. I attached some explanation from Gemini generated from the Minecraft code diff. Basically, client needs no changes for globalIndex, it's for bookkeeping purposes only whereas checksum is generated by client and enforced by the server.
globalIndex
The addition of globalIndex to the ClientboundPlayerChatPacket is a key change for making the chat system more robust. Here's a detailed breakdown of how it works.
What is globalIndex?
The globalIndex is a simple, monotonically increasing integer assigned by the server to every chat message packet it sends to a client.
On the server side (specifically in ServerGamePacketListenerImpl.java), a new counter nextChatIndex has been introduced. Each time the server sends a chat packet to a player, it does the following:
- Gets the current value of
nextChatIndex. - Includes this value in the
ClientboundPlayerChatPacketas theglobalIndex. - Increments
nextChatIndexby one.
This means that from the client's perspective, every single message it receives in its chat feed—whether from a player, the system, or a death message—now has a unique, sequential ID.
How is it different from the existing index?
This is the most important distinction to understand. The packet now contains two indices, and they serve entirely different purposes.
-
index(fromSignedMessageLink):- Scope: Per-player.
- Managed by: The sending client.
- Purpose: Cryptographic Chaining and Security. This index is part of the secure chat system. Each player maintains their own message counter. When they send a message, they include their current index and sign it. This allows the server and other clients to verify that the message hasn't been tampered with and that it's part of an unbroken sequence from that specific player. It's for message authenticity.
-
globalIndex:- Scope: Per-receiver (the client's view of chat).
- Managed by: The server.
- Purpose: Message Management and Identification. This index is used by the server to give the client a simple, unambiguous way to refer to a specific message later. Its primary function is to make deleting messages reliable. It's for message identification.
Why was globalIndex added?
The globalIndex solves a critical problem: how to reliably and uniquely identify a message for deletion.
Previously, the ClientboundDeleteChatPacket had to identify a message using its MessageSignature. This had several drawbacks:
- System Messages: System messages (like "Player has joined the game" or command output) don't have a cryptographic signature, so they couldn't be easily targeted for deletion.
- Ambiguity: While unlikely, it's theoretically possible for signature collisions or other edge cases to create ambiguity.
- Complexity: Relying on a complex cryptographic signature for a simple identification task is overkill.
The globalIndex provides a much simpler and more robust solution.
How it Works in Practice (Example Scenario)
Imagine you are a client connected to a server. Here's what your chat log might look like with the new system:
-
Server sends a welcome message:
- Packet:
ClientboundSystemChatPacket(noglobalIndexhere, it's part of the player chat packet) - let's assume the server processes this and wants to show it to you. Correction: The server would wrap this in a player-chat-like structure to send to the client. Let's adjust the example. The server sends chat packets that result in these lines appearing on your screen.
- Packet:
-
PlayerA sends "Hi everyone!"
- Server sends you
ClientboundPlayerChatPacketwith:globalIndex: 101sender: PlayerAindex: 5(PlayerA's 5th message)
- Server sends you
-
You die to a zombie.
- Server sends you a system message packet representing the death message.
- It might be assigned
globalIndex: 102.
-
A moderator wants to delete PlayerA's message.
- The server now has a perfect, unambiguous "handle" for that message. It can send a new (or modified)
ClientboundDeleteChatPacketthat simply says: "Delete message withglobalIndex101." - The client receives this, finds the message in its chat history with
globalIndex101, and removes it. There's no need to parse content or compare complex signatures.
- The server now has a perfect, unambiguous "handle" for that message. It can send a new (or modified)
Summary
In short, the globalIndex is a server-side serial number for chat messages sent to a client. It was added to provide a simple and foolproof way for the server to identify and manage messages on the client side, primarily for making message deletion robust and reliable for all types of chat content. It works alongside the existing per-player index, which is used for cryptographic security.
checksum
Of course! The new checksum byte is a clever and efficient mechanism for ensuring data integrity within the secure chat system.
What the Checksum Does (The 'Why')
The primary purpose of the checksum is to detect desynchronization between the client and the server.
In the secure chat system, the client periodically sends a LastSeenMessages.Update packet to the server. This packet tells the server, "Here is the list of recent messages I have seen and acknowledged." The server uses this information to validate the cryptographic chain of subsequent messages from the player.
However, a problem can arise if the client's and server's understanding of the "list of recent messages" diverges. This is called a desync. For example:
- The client might have missed a message packet.
- The server might have processed messages in a slightly different order.
If a desync occurs, the client's acknowledgement update becomes meaningless. The client might say "I've seen message #5," but on the server, message #5 could be a completely different message. This would cause the server to incorrectly validate or invalidate future chat messages.
The checksum byte solves this problem by acting as a lightweight "fingerprint" of the client's list of last seen messages.
Here's the workflow:
- The client generates the list of messages it has seen (
LastSeenMessages). - It computes a checksum byte based on this list.
- It sends the
LastSeenMessages.Updatepacket, which contains the list of acknowledgements and the checksum. - The server receives the update. Before applying it, it reconstructs what it believes the client's last seen message list should be and computes its own checksum.
- It compares the checksum it just computed with the checksum received from the client.
- If they match: Everything is in sync. The server can trust the acknowledgement update and proceeds.
- If they do not match: A desync has occurred. The server knows it cannot trust the client's chat state and disconnects the player with the error message: "Checksum mismatch on last seen update: the client and server must have desynced". This is a fail-safe to prevent the secure chat system from malfunctioning.
There's one special case: a checksum of 0 is ignored, likely for backward compatibility or special packet handling.
How the Checksum is Generated (The 'How')
The checksum is generated through a simple, deterministic hashing process. Here is the step-by-step breakdown based on the code changes:
Step 1: The Checksum of a Single Message Signature
The fundamental building block is the checksum of an individual MessageSignature. A new method checksum() was added to MessageSignature.java:
// in MessageSignature.java
int checksum() {
return Arrays.hashCode(this.bytes);
}This uses Java's standard Arrays.hashCode() on the raw byte array of the signature. It produces a consistent int hash for any given signature.
Step 2: Aggregating into a List Checksum
Next, the LastSeenMessages class (which is essentially a list of signatures) aggregates these individual checksums into one. A new method computeChecksum() was added:
// in LastSeenMessages.java
byte computeChecksum() {
int b = 1; // Initial value
for(MessageSignature d : this.entries) {
b = 31 * b + d.checksum(); // Classic hashing pattern
}
byte e = (byte)b;
return e == 0 ? 1 : e; // Avoids the special '0' value
}This method iterates through all the message signatures in its list and combines their individual checksums using a standard hashing algorithm (hash = 31 * hash + new_element_hash).
Step 3: Converting to a Byte and Sending
- The final
intresult from the loop is cast to abyte. This simply takes the lowest 8 bits of the integer, which is a fast way to create a small checksum. - The code explicitly checks if the resulting byte is
0. Since0is used as a special value to ignore the checksum, the method ensures a valid checksum can never be zero by changing it to1if it is. - This final byte is then included in the
LastSeenMessages.Updatepacket and sent from the client to the server for validation.
In summary, the checksum is a hash of hashes—an aggregation of the hash codes of all message signatures in the client's "last seen" list. It's a very fast and efficient way for the server to confirm that its view of the client's chat history is perfectly in sync, making the entire secure chat system more reliable.
| function calculateChecksum (messages) { | ||
| if (!messages || messages.length === 0) return 0 | ||
|
|
||
| let checksum = 0 | ||
| for (const msg of messages) { | ||
| if (msg.signature) { | ||
| // XOR the first byte of each signature for a simple checksum | ||
| checksum ^= msg.signature[0] || 0 | ||
| } | ||
| } | ||
| return checksum | ||
| } |
There was a problem hiding this comment.
The checksum algorithm doesn't seem to match Minecraft, did you write this intentionally? The server does verify this, so this has to be correct if chat signing is enabled (the default) or the server will kick for CHAT_VALIDATION_FAILED. I attached some logs from Gemini explaining the changes below
resolve #1385
Waiting for PrismarineJS/minecraft-data#995 to be merge for minecraft-data to be up to date