Packet (and other data) extractor

[maciek134]

Moving discussion over from Discord.

The main idea is to write a small program that will take a mapped jar and output packet structure, ids and other useful info and then potentially generate/verify structs in the codebase.

Next step would be to do what the Java mod does because what's required for packet extraction is almost the same as pulling information about blocks, entities etc.

Currently I have a bare-bones fake JVM that can evaluate some bytecode - just enough to get the info we need, it won't call functions or do any logic. This is used to parse packet writer methods and extract packet structure (because class fields are not ordered).

Example output atm (not even close to what's supposed to be output, just a showcase of the state the work is in):

net/minecraft/network/packet/s2c/play/LookAtS2CPacket: [("selfAnchor", Struct("net.minecraft.command.argument.EntityAnchorArgumentType$EntityAnchor")), ("targetX", F64), ("targetY", F64), ("targetZ", F64), ("lookAtEntity", Bool), ("entityId", I32), ("targetAnchor", Struct("net.minecraft.command.argument.EntityAnchorArgumentType$EntityAnchor"))]
ABORTING: method contains branching logic which is not handled yet!
net/minecraft/network/packet/c2s/play/QueryBlockNbtC2SPacket: [("transactionId", I32), ("pos", Struct("net.minecraft.util.math.BlockPos"))]
net/minecraft/network/packet/c2s/play/ClientStatusC2SPacket: [("mode", Struct("net.minecraft.network.packet.c2s.play.ClientStatusC2SPacket$Mode"))]
net/minecraft/network/packet/c2s/play/TeleportConfirmC2SPacket: [("teleportId", I32)]
net/minecraft/network/packet/s2c/play/ItemPickupAnimationS2CPacket: [("entityId", I32), ("collectorEntityId", I32), ("stackAmount", I32)]
net/minecraft/network/packet/s2c/play/NbtQueryResponseS2CPacket: [("transactionId", I32), ("nbt", Struct("net.minecraft.nbt.NbtCompound"))]

This won't be a fully automatic process, but should help with upgrading versions. I'll push the code once I clean it up a bit, this is more for general discussion about it.

cc @JackCrumpLeys since you were interested and this could help you with the 1.21.x work.

[maciek134]

Mappings

Mappings used don't matter, won't matter and I'm not going to change them. But to avoid another lengthy discussion, here are the reasons I'm developing this with yarn and not official mappings (beyond personal preference):

• license - there is no ambiguity about what can and cannot be done with them
• completeness - official mappings don't contain method params which are useful to understand what I'm doing when something breaks
  • I know Parchment exists, but it's scope is too wide and I'd scratch my eyes out looking at the prefixes on all the params
• stability and conventions - official mappings offer no guarantee about anything; yarn mappings avoid changing things arbitrarily and have clear rules about how things should be named
• direction - official mappings come from Mojang/MS, they decide how things are named and that's it; yarn mappings are built by the community and everyone has the ability to get involved

[dyc3]

What's wrong with the current extractor?

[JackCrumpLeys]

The idea is to automatically generate packet structs not just names, ids etc. Docs and smarter encoding will still be done by hand but this removes the bother of redoing hundreds of packets every update.

[maciek134]

@dyc3 Other than it being Java so feeling a bit weird in a Rust project, it's limited by the reflection API - it can't access the content of methods which is necessary to gather packet structures and it already needs a workaround to get tracking data default values.

While it could be done with ASM or something like it, I don't enjoy Java so it made more sense to me to implement it with Rust.

The new one doesn't need to start a client and should work between versions with no or minimal changes too.

[dyc3]

Ah, ok. Makes sense. Very interested to see how this would work.

[JackCrumpLeys]

Are you still working on this? You left the discord server.

[maciek134]

Doing registries today was definitely too optimistic. Java generics suck and took most of my time today, but the result is pretty nice:

net/minecraft/network/packet/s2c/common/CustomReportDetailsS2CPacket: [
    ("details", Map(String, String), "DETAILS_CODEC")
]
net/minecraft/network/packet/s2c/play/ServerMetadataS2CPacket: [
    ("description", Struct("net.minecraft.text.Text"), "PACKET_CODEC"),
    ("favicon", Generic("java.util.Optional", [List(List(U8))]), "unknown")
]
net/minecraft/network/packet/s2c/custom/DebugBrainCustomPayload$Brain: [
    ("uuid", Struct("java.util.UUID"), None, false),
    ("entityId", I32, None, false),
    ("name", String, None, false),
    ("profession", String, None, false),
    ("experience", I32, None, false),
    ("health", F32, None, false),
    ("maxHealth", F32, None, false),
    ("pos", Struct("net.minecraft.util.math.Vec3d"), None, false),
    ("inventory", String, None, false),
    ("path", Struct("net.minecraft.entity.ai.pathing.Path"), None, false),
    ("wantsGolem", Bool, None, false),
    ("angerLevel", I32, None, false),
    ("possibleActivities", List(String), None, false),
    ("runningTasks", List(String), None, false),
    ("memories", List(String), None, false),
    ("gossips", List(String), None, false),
    ("pois", Set(Struct("net.minecraft.util.math.BlockPos")), None, false),
    ("potentialPois", Set(Struct("net.minecraft.util.math.BlockPos")), None, false),
]
net/minecraft/network/packet/s2c/play/ChatSuggestionsS2CPacket: [
    ("action", Struct("net.minecraft.network.packet.s2c.play.ChatSuggestionsS2CPacket$Action"), None, false),
    ("entries", List(String), None, false),
]

First two are parsed from codec declarations, next two from a write method.
Codecs still need more work too, sometimes writes go through multiple lambdas and interfaces and I'm currently stopping one-deep.
I'm currently not handling bitflags automatically and won't in the initial implementation, but should be easy in the future - it's usually a series of "read boolean field → if false jump over → bitwise OR on a local variable".
Inner classes (like ChatSuggestionsS2CPacket$Action) might need special handling, but in general it can lookup all loaded classes as needed.

[JackCrumpLeys]

How is this going?

[maciek134]

Got stuck on one thing like two weeks ago and my ADHD took me on a side quest. I'm back working on this now, but I'm traveling on the weekend so not sure I'll have anything worth updating about until next week.

[JackCrumpLeys]

Hey, I can understand if you're no longer motivated to work on this. But development is starting back up again, and this would be useful to have. I would like the code, if that's okay. I'm fine if it's not in a compiling state :).

[maciek134]

Hey, yeah I decided I'm not going to spend time on a server just to be on the mercy of M$ deciding what's allowed or not. I'll push what I had after work today, just keep in mind it's basically a playground I was trying stuff in, so the code is a huge mess. Though iirc it should compile