[Bug] Avro bytes fields rendered as Latin-1 codepoint strings instead of Base64 since v3.7.0

[haoyukongTrackunit]

Description

Since v3.7.0, Avro bytes fields — including those inside Debezium logical types such as io.debezium.data.VariableScaleDecimal — are rendered in the Console UI as raw Latin-1/UTF-8 codepoint strings (e.g. "\r\u0003\u0012\u00aa\u00a3_\u00f2") instead of Base64 (e.g. "QwhcN9vr+A==").

This is a regression from v2.8.x behaviour and breaks the ability to visually verify decimal values encoded via Debezium's VariableScaleDecimal type directly in the Console UI.

Root Cause

The regression was introduced in PR #2351 (commit 14d05a16f), which changed Avro serialisation from Go's json.Marshal to avroSch.EncodeJSON. The Avro JSON spec encodes bytes as ISO-8859-1 codepoint strings rather than Base64. While spec-compliant, this is a breaking behaviour change for consumers who relied on the previous Base64 output for manual data verification.

Steps to Reproduce

1. Configure a Kafka topic with Avro-encoded messages containing bytes fields, or Debezium CDC messages containing io.debezium.data.VariableScaleDecimal fields (e.g. a numeric/decimal PostgreSQL column via Debezium).
2. Open the topic in Redpanda Console v3.7.0 or v3.7.1.
3. Inspect a message — the bytes / VariableScaleDecimal.value field renders as a raw codepoint string instead of Base64.

Expected Behaviour

bytes fields should be rendered as Base64-encoded strings, consistent with v2.8.x behaviour.

v2.8.11 output (expected):

"cumulative_emissions": {
  "io.debezium.data.VariableScaleDecimal": {
    "scale": 15,
    "value": "cVLOXkWiqq=="
  }
}

Actual Behaviour

v3.7.1 output (actual):

"cumulative_emissions": {
  "scale": 13,
  "value": "\tn\\\u001a#\u00dbN"
}

Bytes above 0x7F are corrupted when re-encoded as UTF-8 codepoints, making the original value unrecoverable from the UI alone.

Impact

1. Lossless verification broken — Base64 can be reliably decoded to the original byte sequence; Latin-1 codepoint strings mangle bytes above 0x7F and the original data cannot be recovered from the UI.
2. Manual data validation impossible — teams using Console to spot-check CDC pipeline values (e.g. Databricks → PostgreSQL → Debezium → Redpanda) can no longer verify decimal field values without accessing raw Avro binaries or a separate toolchain.
3. AI/tooling decode failure — even LLM-based tools cannot correctly decode the mangled codepoint representation because the original bytes are already lost in the re-encoding.
4. Silent corruption risk — data that looks like text is actually mangled bytes, making it harder to detect real pipeline issues early.

Environment

• Redpanda Console versions affected: v3.7.0, v3.7.1 (reproduced on both)
• Last working version: v2.8.11
• Schema type: Debezium PostgreSQL CDC, io.debezium.data.VariableScaleDecimal logical type
• Schema Registry: Confluent-compatible (kafka-cp-schema-registry)

Notes

• The change in PR serde: use EncodeJSON and DecodeJSON for Avro serialization #2351 was intentional and also fixed real bugs (NaN/±Inf serialisation errors, wrong time.Duration units, union wrappers on decode). A naive revert is not recommended.
• Suggested fix: keep EncodeJSON but post-process []byte/fixed Avro fields back to Base64 in the UI-facing JSON payload.
• v3.6.0 does not exhibit this specific bytes regression but has other known issues (schema subject resolution), so downgrading to v3.6.0 is not a viable workaround.
• Verified by locally running Console v2.8.11, v3.6.0, and v3.7.1 side-by-side against the same production Kafka cluster and schema registry.

[haoyukongTrackunit]

[twmb]

Thanks for the detailed report and the side-by-side screenshots. The UI c&p problem can be solved. I don't think we should revert back to base64 encoding.

The wire payload is not corrupted.
EncodeJSON writes bytes per the Avro JSON spec: every byte > 0x7F becomes a \uXXXX escape, and the sequence is byte-exact 1:1 reversible. Round-trip is lossless (twmb/avro has round-trip tests covering EncodeJSON -> DecodeJSON).

The loss you're seeing is in the frontend JSON viewer.
After the JSON text arrives, it passes through JSON.parse (turning Û into the JS string character U+00DB) and then JSON.stringify for display. JSON.stringify doesn't re-escape non-ASCII, so U+00DB renders as the glyph Û, and "Copy Value" places UTF-8 bytes (C3 9B) on the clipboard instead of DB. This is a regression.

Fix: frontend-only, scoped to Avro payloads. The JSON viewer re-escapes code points in 0x80-0xFF back to \u00XX before display. After this, the UI shows the same string that's on the wire, and copy-paste recovers the original bytes with any Avro-JSON-aware decoder (one line of Python / Go / whatever). No effect on non-Avro messages.

Base64 wasn't exactly intentional, it was a side effect of running json.Marshal and the old path was silently producing wrong output for several Avro types.

• time.Duration with time-millis/time-micros logical types: written as nanoseconds. A 100ms field displayed as 100000000 (nanoseconds) instead of 100 (millis).
• Decimal logical types (*big.Rat): written as "numerator/denominator" ("355/113") via big.Rat.MarshalText (not round-trippable)
• Fixed types: stdlib only base64-encodes []byte slices. Fixed [N]byte arrays marshal as JSON arrays of integers ([9, 26, 219, 78]). A bytes field and a fixed field carrying the same data rendered as completely different shapes in the same record.
• Timestamp / date / time-of-day fields: written as RFC3339 strings ("2026-04-22T12:23:31Z"). The Avro JSON spec expects an integer (millis, micros, or days since epoch), so the output was human-readable but the wrong shape for any other Avro JSON consumer, and it hid the schema's actual precision.
• NaN / ±Infinity floats: json.Marshal errored out, so the entire record failed to display.

EncodeJSON is schema-aware and handles all of these per the spec. The old path had a bunch of silent corruption.

On the specific points in your "Impact" section:

1. Lossless verification: preserved on the wire; restored in the UI copy-paste path by the fix above.
2. Manual data validation: still doable via any Avro-JSON decoder (Go: twmb/avro DecodeJSON; Python one-liner below):

   python3 -c 'import sys,json,base64; print(base64.b64encode(bytes(map(ord, json.loads(sys.stdin.read())))).decode())'

3. AI/tooling decode: any Avro-JSON-spec-aware decoder recovers bytes exactly (twmb/avro's DecodeJSON does this).
4. Silent corruption risk: inverted. The old path silently corrupted time.Duration, decimal, timestamps, and fixed types; EncodeJSON doesn't.

PR #2425 can address the display-copy-and-paste issue. What do you think?

[haoyukongTrackunit]

Thanks for the quick response and fix PR. Using

as an example, as long as we do not get Ñ or º instead of \u00d1 and \u00ba, and invisible bytes like \u009a are no longer silently dropped on copy-paste, then we can convert it to decimal, I am good with the fix PR. Thanks!