[Bug]: ByteBuddy doesn't do well with Map<T, ByteBuffer>

[ahmedabu98]

What happened?

I have an @autovalue class with the @DefaultSchema(AutoValueSchema.class). One of the getters/setters is a Map<Integer, ByteBuffer>. When using elements of this class in my pipeline, I get the following error:

Error stack trace

java.lang.ClassCastException: class [B cannot be cast to class java.nio.ByteBuffer ([B and java.nio.ByteBuffer are in module java.base of loader 'bootstrap')
	at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Function$SchemaCodeGen$sIwEd1pP.apply(Unknown Source)
	at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils$TransformingMap.<init>(ByteBuddyUtils.java:525)
	at org.apache.beam.sdk.schemas.utils.ByteBuddyUtils.getTransformingMap(ByteBuddyUtils.java:508)
	at org.apache.beam.sdk.io.iceberg.FieldValueGetter$SchemaCodeGen$c6I5wjZP.get(Unknown Source)
	at org.apache.beam.sdk.io.iceberg.FieldValueGetter$SchemaCodeGen$c6I5wjZP.get(Unknown Source)
	at org.apache.beam.sdk.values.RowWithGetters$1.apply(RowWithGetters.java:82)
	at org.apache.beam.sdk.values.RowWithGetters$1.apply(RowWithGetters.java:75)
	at java.base/java.util.Map.computeIfAbsent(Map.java:1003)
	at org.apache.beam.sdk.values.RowWithGetters.getValue(RowWithGetters.java:73)
	at org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:419)
	at org.apache.beam.sdk.coders.Coder$ByteBuddy$w90nPOHK.encode(Unknown Source)
	at org.apache.beam.sdk.coders.Coder$ByteBuddy$w90nPOHK.encode(Unknown Source)
	at org.apache.beam.sdk.schemas.SchemaCoder.encode(SchemaCoder.java:121)
	at org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:437)
	at org.apache.beam.sdk.coders.Coder$ByteBuddy$1ANxtLai.encode(Unknown Source)
	at org.apache.beam.sdk.coders.Coder$ByteBuddy$1ANxtLai.encode(Unknown Source)
	at org.apache.beam.sdk.schemas.SchemaCoder.encode(SchemaCoder.java:121)
	at org.apache.beam.sdk.coders.Coder.encode(Coder.java:135)
	at org.apache.beam.sdk.util.CoderUtils.encodeToSafeStream(CoderUtils.java:86)
	at org.apache.beam.sdk.util.CoderUtils.encodeToByteArray(CoderUtils.java:70)
	at org.apache.beam.sdk.util.CoderUtils.encodeToByteArray(CoderUtils.java:55)
	at org.apache.beam.sdk.util.CoderUtils.clone(CoderUtils.java:168)
	at org.apache.beam.runners.direct.CloningBundleFactory$CloningBundle.add(CloningBundleFactory.java:87)
	at org.apache.beam.runners.direct.FlattenEvaluatorFactory$FlattenEvaluator.processElement(FlattenEvaluatorFactory.java:75)
	at org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:165)
	at org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:129)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

I think I was able to track the problem down to this code in ByteBuddy:

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/ByteBuddyUtils.java

Lines 1113 to 1125 in e4ab66d

	@Override
	protected StackManipulation convertMap(TypeDescriptor<?> type) {
	Type rowKeyType =
	getFactory().createTypeConversion(false).convert(ReflectUtils.getMapType(type, 0));
	final TypeDescriptor keyElementType = ReflectUtils.getMapType(type, 0);
	Type rowValueType =
	getFactory().createTypeConversion(false).convert(ReflectUtils.getMapType(type, 1));
	final TypeDescriptor valueElementType = ReflectUtils.getMapType(type, 1);
	StackManipulation readTrasformedValue = readValue;
	if (!keyElementType.hasUnresolvedParameters()
	&& !valueElementType.hasUnresolvedParameters()) {
	ForLoadedType keyConversionFunction =

From what I understand, Beam does some conversions between ByteBuffer and byte[] (for reasons I am not aware of). In the above code, the Map's key and value element types are not transparent. Instead, they show up as generics. Because of this, we never enter the if block, which contains code that applies conversion logic.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[reuvenlax]

This may fixed now that we are resolving all generic types

[ahmedabu98]

This is indeed fixed by both #32757 and #32810. Will close this issue after the latter is merged

[Abacn]

I saw #32757 was reverted in #32887, is this Issue still at resolved stage? For same reason, is #32821 ready to get in?

[ahmedabu98]

Reopening because we decided to go with the solution in #32081

[damccorm]

Looks like this got the 2.61.0 milestone added when it was closed/reopened; should this be a release blocker?

[ahmedabu98]

Not a blocker, removed the milestone