Skip to content

Encoding: protojson.Marshal are not determenistic and may return unstable results #116

@ciricc

Description

@ciricc
// Marshal marshals the given [proto.Message] in the JSON format using options in
// Do not depend on the output being stable. Its output will change across
// different builds of your program, even when using the same version of the
// protobuf module.
func (o MarshalOptions) Marshal(m proto.Message) ([]byte, error) {
	return o.marshal(nil, m)
}

We can use some hack like this:

protojson.MarshalOptions{
	Indent: " ", // Non-empty indent disables detrand whitespace
}.Marshal(output)

This configuration allows us to skip the adding whitespace before the next field name during encoding, but i think that the most right way to fix the problem - use another protojson encoder for determinitstic results. Thats because of the lines: https://github.com/protocolbuffers/protobuf-go/blob/d65e1d490c91cecb040049dd09d1ac866bc2ce3a/internal/encoding/text/encode.go#L224
and the detrand usage.

I've tested the workflow written on the golang with this encoder and i am getting sometimes this problem during new releases.

ERROR Invocation panicked, returning error to Restate method=workflow.video.IndexVideoWorkflow/Run invocationID=inv_1fKqBMJteeH355lZHzuQSqwTN5GH4nvPqM err="[570] [570] (570) Found a mismatch between the code paths taken during the previous execution and the paths taken during this execution.\nThis typically happens when some parts of the code are non-deterministic.\n- The mismatch happened while executing 'call' (index '5')\n- Difference:\n parameter: '{\"audioTrackId\":{\"value\":\"2d15d8c2-8675-464f-a809-87248baa173f\"},\"options\":{\"wordCountThreshold\":8}}' != '{\n \"audioTrackId\": {\n \"value\": \"2d15d8c2-8675-464f-a809-87248baa173f\"\n },\n \"options\": {\n \"wordCountThreshold\": 8\n }\n}'\nRelated command: call [5]"

Alternative solution (officially by the protobuf team):

"To gain some degree of output stability, we recommend running the output through a JSON formatter."

Source: https://protobuf.dev/reference/go/faq/#unstable-json

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions