Jackson subtype Avro schema unions are non-deterministic and therefore incompatible with each other

[raphw]

As a consequence of this change, the order of schema formats is no longer deterministic: 0d7f007 (see #589 / #593 )

This means that two schemas that are generated in two separate runs are incompatible. The reason for this is the use of a Set for collecting unionSchemas. To solve the issue, there should be a separate list where types are stored in the discovery order to retain a deterministic order.

[beseder1]

The issue is currently blocking us severely using 2.19.2. The fix should probably be rather simple as sketched below. Is this something you will be pick up soon for 2.19.3/2.20.1 or should we raise a PR? I am asking because we haven't contributed so far.

public RecordVisitor(SerializerProvider p, JavaType type, VisitorFormatWrapperImpl visitorWrapper)
    {
        super(p);
        _type = type;
        _visitorWrapper = visitorWrapper;
        // Check if the schema for this record is overridden
        BeanDescription bean = getProvider().getConfig().introspectDirectClassAnnotations(_type);
        AvroSchema ann = bean.getClassInfo().getAnnotation(AvroSchema.class);
        if (ann != null) {
            _avroSchema = AvroSchemaHelper.parseJsonSchema(ann.value());
            _overridden = true;
            _typeSchema = null;
        } else {
            // If Avro schema for this _type results in UNION I want to know Avro type where to assign fields
            _avroSchema = AvroSchemaHelper.initializeRecordSchema(bean);
            _typeSchema = _avroSchema;
            _overridden = false;
            AvroMeta meta = bean.getClassInfo().getAnnotation(AvroMeta.class);
            if (meta != null) {
                _avroSchema.addProp(meta.key(), meta.value());
            }

            List<NamedType> subTypes = getProvider().getAnnotationIntrospector().findSubtypes(bean.getClassInfo());
            if (subTypes != null && !subTypes.isEmpty()) {
                // alreadySeenClasses prevents subType processing in endless loop
                Set<Class<?>> alreadySeenClasses = new HashSet<>();
                alreadySeenClasses.add(_type.getRawClass());

                // At this point calculating hashCode for _typeSchema fails with
                // NPE because RecordSchema.fields is NULL
                // (see org.apache.avro.Schema.RecordSchema#computeHash).
                // Therefore, unionSchemas must not be HashSet (or any other type
                // using hashCode() for equality check).
                // ArrayList ensures that ordering of subTypes is preserved.
                final List<Schema> unionSchemas = new ArrayList<>();
                // Initialize with this schema
                if (_type.isConcrete()) {
                    unionSchemas.add(_typeSchema);
                }

                try {
                    for (NamedType subType : subTypes) {
                        if (!alreadySeenClasses.add(subType.getType())) {
                            continue;
                        }
                        JsonSerializer<?> ser = getProvider().findValueSerializer(subType.getType());
                        VisitorFormatWrapperImpl visitor = _visitorWrapper.createChildWrapper();
                        ser.acceptJsonFormatVisitor(visitor, getProvider().getTypeFactory().constructType(subType.getType()));
                        // Add subType schema into this union, unless it is already there.
                        Schema subTypeSchema = visitor.getAvroSchema();
                        // When subType schema is union itself, include each its type into this union if not there already
                        if (subTypeSchema.getType() == Type.UNION) {
                            unionSchemas.addAll(subTypeSchema.getTypes());
                        } else {
                            unionSchemas.add(subTypeSchema);
                        }
                    }
                    _avroSchema = Schema.createUnion(deduplicateByReference(unionSchemas));
                } catch (JsonMappingException jme) {
                    throw new RuntimeJsonMappingException("Failed to build schema", jme);
                }
            }
        }
        _visitorWrapper.getSchemas().addSchema(type, _avroSchema);
    }

    private static List<Schema> deduplicateByReference(List<Schema> schemas) {
        List<Schema> result = new ArrayList<>();
        // IdentityHashMap is used because we need to deduplicate by reference.
        Map<Schema, Boolean> seen = new IdentityHashMap<>();

        for(Schema s : schemas) {
            if(!seen.containsKey(s)) {
                seen.put(s, Boolean.TRUE); // mark as seen (by reference)
                result.add(s);             // preserve order
            }
        }
        return result;
    }

[cowtowncoder]

*beseder1* left a comment (FasterXML/jackson-dataformats-binary#601) <#601 (comment)> The issue is currently blocking us severely using 2.19.2. The fix should probably be rather simple as sketched below. Is this something you will be pick up soon for 2.19.3/2.20.1 or should we raise a PR? I am asking because we haven't contributed so far.

Raising a pr would be very helpful if you could do that! I am happy to help — could possibly go against 2.19 if safe enough. Although probably best to do PR against `2.x` and I can decide where to backport, based on likely risk (my earlier comment referenced 2.18 but earlier change was for 2.19 so ignore that).

[beseder1]

@cowtowncoder Thank your response! I will provide a PR soon and contact you if questions appear.

[beseder1]

Following your comment I just created one against 2.x in my fork.