-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Closed
Description
What happened?
When type is read from an avro, (see example below).
...
"name": "a_name",
"type": [
"null",
{
"avro.java.string": "String",
"type": "string"
}
]
...
This will fail with the following error:
INFO 2025-06-24T08:00:07.685658Z >> ReadFromAvro(f"{input_path}.avro", as_rows=True)
INFO 2025-06-24T08:00:07.685908Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO 2025-06-24T08:00:07.686203Z File "/usr/src/app/.venv/lib/python3.12/site-packages/apache_beam/io/avroio.py", line 170, in __init__
INFO 2025-06-24T08:00:07.686475Z beam_schema = avro_schema_to_beam_schema(avro_schema)
INFO 2025-06-24T08:00:07.686678Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO 2025-06-24T08:00:07.686897Z File "/usr/src/app/.venv/lib/python3.12/site-packages/apache_beam/io/avroio.py", line 607, in avro_schema_to_beam_schema
INFO 2025-06-24T08:00:07.687135Z beam_type = avro_type_to_beam_type(avro_schema)
INFO 2025-06-24T08:00:07.687346Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO 2025-06-24T08:00:07.687559Z File "/usr/src/app/.venv/lib/python3.12/site-packages/apache_beam/io/avroio.py", line 598, in avro_type_to_beam_type
INFO 2025-06-24T08:00:07.687771Z f['name'], avro_type_to_beam_type(f['type']))
INFO 2025-06-24T08:00:07.687972Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO 2025-06-24T08:00:07.688190Z File "/usr/src/app/.venv/lib/python3.12/site-packages/apache_beam/io/avroio.py", line 576, in avro_type_to_beam_type
INFO 2025-06-24T08:00:07.688387Z return avro_union_type_to_beam_type(avro_type)
INFO 2025-06-24T08:00:07.688579Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO 2025-06-24T08:00:07.688775Z File "/usr/src/app/.venv/lib/python3.12/site-packages/apache_beam/io/avroio.py", line 563, in avro_union_type_to_beam_type
INFO 2025-06-24T08:00:07.688998Z if avro_type in AVRO_PRIMITIVES_TO_BEAM_PRIMITIVES:
INFO 2025-06-24T08:00:07.689308Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO 2025-06-24T08:00:07.689545Z TypeError: unhashable type: 'dict'
I was able to reproduce this in a unit test.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Reactions are currently unavailable