Polymophism with @XmlValue and types with identical tags

[garrison-henkle]

I've been trying to represent an EMV XML file that looks like this using the library:

<EmvTag tag="77">
    <EmvTag tag="9f15">4123</EmvTag>
    <EmvTag tag="5f28>840</EmvTag>
</EmvTag>

I originally tried building a sealed hierarchy for this:

sealed interface Tag {
    val tag: String

    @Serializable
    @SerialName(value = "EmvTag")
    data class Primitive(
        override val tag: String,
        @XmlValue
        val value: String,
    ) : Tag

    @Serializable
    @SerialName(value = "EmvTag")
    data class Constructed(
        override val tag: String,
        val children: List<Tag>,
    ) : Tag
}

but both types having the same tag seems to break things:

Multiple sealed subclasses of 'class Tag' have the same serial name 'EmvTag': 'class Tag$Constructed', 'class Tag$Primitive'

I tried adding a custom serializer to work around this:

class Serializer : KSerializer<Tag> {
    override val descriptor: SerialDescriptor = buildClassSerialDescriptor("Tag")

    override fun serialize(encoder: Encoder, value: Tag) {
        when (value) {
            is Tag.Primitive -> {
                encoder.encodeSerializableValue(
                    serializer = Tag.Primitive.serializer(),
                    value = value,
                )
            }
            is Tag.Constructed -> {
                encoder.encodeSerializableValue(
                    serializer = Tag.Constructed.serializer(),
                    value = value,
                )
            }
        }
    }

    override fun deserialize(decoder: Decoder): Tag {
        val xmlDecoder = decoder as? XML.XmlInput ?: error("Serializer only supports XML")
        val reader = xmlDecoder.input

        return if (isConstructed(reader.getAttributeValue("", "tag") ?: "")) {
            Tag.Constructed.serializer().deserialize(decoder)
        } else {
            Tag.Primitive.serializer().deserialize(decoder)
        }
    }

    private fun isConstructed(tag: String): Boolean =
        try {
            val firstByte = tag.substring(0, 2).toInt(16)
            (firstByte and 0x20) != 0
        } catch (_: Exception) {
            false
        }
}

but the lack of a real descriptor broke things again:

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266)

I tried to add a elements to the descriptor for tag, children (optional), and value (optional), but referencing List<Tag>> in the definition creates a loop that causes a stack overflow. I also tried replacing List<Tag>> with the descriptor for a delegate class that had an identical structure, but it still crashed.

Finally, I also tried to flatten the data structure into something like this:

@Serializable
@SerialName(value = "EmvTag")
data class Tag(
    override val tag: String,
    val children: List<Tag> = emptyList(),
    @XmlValue
    val value: String? = null,
) : Tag

But this also fails since @XmlValue can't be used with another property that parses child elements.

Is there a supported way to parse something like this where one of the polymorphic subclasses needs to use @XmlValue and both subclasses need to serialize to the same tag name? I've been able to get just deserialization working with a partially manual serializer (I'll put it below), but I can't figure out a way to get both directions working correctly. The errors I've been getting also make perfect sense to me, but I seem to discover new errors with every attempt at a workaround 😅 . Sorry if this is a bit of a dumb question–I've tried much more than I've mentioned here and don't really know what else to try at this point. Neither the polymorphic examples in the repo nor the tests seem to cover this edge case as far as I can tell.

The manual deserializer:

override fun deserialize(decoder: Decoder): EmvTagValue<T> {
        val xmlDecoder = decoder as? XML.XmlInput ?: throw Exception("Only XML supported")
        val reader = xmlDecoder.input

        // the reader can have isStarted=false if it a self-closing tag e.g. <EmvTag tag="70"/>
        if (!reader.isStarted || reader.eventType == EventType.END_ELEMENT) {
            return EmvTagValue.Tags(tags = emptyList())
        }

        val xml = XML(config = xmlDecoder.config, serializersModule = xmlDecoder.serializersModule)

        while (reader.isWhitespace() && reader.hasNext()) {
            reader.next()
        }

        return when (reader.eventType) {
            EventType.TEXT, EventType.CDSECT -> {
                val text = reader.text
                reader.next()
                EmvTagValue.Text(value = text)
            }

            EventType.START_ELEMENT -> {
                val tags = mutableListOf<T>()

                while (reader.eventType == EventType.START_ELEMENT && reader.localName == tagName) {
                    tags += xml.decodeFromReader(reader = reader, deserializer = valueSerializer)

                    if (reader.eventType == EventType.END_ELEMENT && reader.hasNext()) {
                        reader.next()
                    }

                    while (reader.isWhitespace() && reader.hasNext()) {
                        reader.next()
                    }
                }

                EmvTagValue.Tags(tags = tags)
            }

            EventType.END_ELEMENT -> EmvTagValue.Tags(tags = emptyList())

            else ->
                throw SerializationException(
                    message = "Unexpected element type ${reader.eventType}!",
                )
        }
    }

[pdvrieze]

Have a look at TestMixedTypesafe and MixedContent

These do more or less what you want (except it uses a list). You should probably be able to use MixedContent as is, but it is also helpful as an example of a more appropriate implementation of your case.

[pdvrieze]

Those classes use open polymorphism. In your case you can possibly use sealed polymorphism (or use XmlSerializer for special handling of XML content - not unlike ContentPolymorphicSerializer in Json, but it works rather different)

[garrison-henkle]

I was able to get something working using XmlSerializer, thanks! Serialization with the MixedContent and custom MixedContent-like serializers worked fine, but I kept running into issues during deserialization since the polymorphic serializer didn't like both EmvTag.Primitive and EmvTag.Constructed having the same tag.

In case anyone else comes across this, here is my final implementation that works with both XML and JSON (JSON needed a delegate since @KeepGeneratedSerializer doesn't work on sealed hierarchies/abstract classes/etc.):

@Serializable(with = ManualEmvTagSerializer::class)
sealed interface EmvTag {
    val tag: String

    @Serializable
    @XmlSerialName(value = "EmvTag")
    data class Primitive(
        override val tag: String,
        @XmlValue(value = true)
        val value: String
    ) : EmvTag

    @Serializable
    @XmlSerialName(value = "EmvTag")
    data class Constructed(
        override val tag: String,
        @XmlValue(value = true)
        val children: List<EmvTag>
    ) : EmvTag
}

@Serializable
data class EmvTagJsonDelegate(
    val tag: String,
    val value: String? = null,
    val children: List<EmvTag>? = null,
) {
    fun asEmvTag(): EmvTag =
        if (tag.isConstructed()) {
            EmvTag.Constructed(tag = tag, children = children ?: emptyList())
        } else {
            EmvTag.Primitive(tag = tag, value = value ?: "")
        }
}

@OptIn(ExperimentalXmlUtilApi::class)
data object ManualEmvTagSerializer : AbstractXmlSerializer<EmvTag>() {
    private const val TAG_NAME = "EmvTag"
    private const val ATTR_TAG = "tag"

    private val delegate = PolymorphicSerializer(baseClass = Any::class)
    override val descriptor: SerialDescriptor = delegate.descriptor

    override fun serializeXML(
        encoder: Encoder,
        output: XmlWriter,
        value: EmvTag,
        isValueChild: Boolean,
    ) {
        with (receiver = output) {
            startTag(namespace = "", localName = TAG_NAME, prefix = "")
            attribute(namespace = "", name = ATTR_TAG, prefix = "", value = value.tag)
            when (value) {
                is EmvTag.Primitive -> text(text = value.value)
                is EmvTag.Constructed -> {
                    for (child in value.children) {
                        serializeXML(
                            encoder = encoder,
                            output = output,
                            value = child,
                            isValueChild = true,
                        )
                    }
                }
            }
            endTag(namespace = null, localName = TAG_NAME, prefix = null)
        }
    }

    override fun deserializeXML(
        decoder: Decoder,
        input: XmlReader,
        previousValue: EmvTag?,
        isValueChild: Boolean,
    ): EmvTag =
        with (receiver = input) {
            val tags = Stack<String>()
            val values = Stack<String>()
            val children = Stack<MutableList<EmvTag>>()
            val constructed = Stack<Boolean>()

            while (true) {
                when (eventType) {
                    EventType.START_ELEMENT -> {
                        if (localName != TAG_NAME) {
                            error(msg = "Expected $TAG_NAME but found $localName!")
                        }

                        if (constructed.peek() == false) {
                            error(msg = "Unexpected element $localName nested within presumed Primitive tag!")
                        }

                        attributes.firstOrNull { it.localName == ATTR_TAG }?.value?.let { value ->
                            tags.push(element = value)
                            val isConstructed = value.isConstructed()
                            constructed.push(element = isConstructed)
                            if (isConstructed) {
                                children.push(element = mutableListOf())
                            }
                        } ?: error(msg = "Missing tag attribute on EmvTag!")
                    }

                    EventType.END_ELEMENT -> {
                        if (localName != TAG_NAME) {
                            error(msg = "Expected $TAG_NAME but found $localName!")
                        }

                        val isConstructed = constructed.pop()
                            ?: error(msg = "Found an unexpected $localName end element!")
                        val tagName = tags.pop()
                            ?: error(msg = "Missing tag for ${if (isConstructed) "Constructed" else "Primitive"} tag!")
                        val isChild = constructed.peek() == true
                        val tag = if (isConstructed) {
                            val children = children.pop() ?: error(msg = "Missing children list for Constructed tag!")
                            EmvTag.Constructed(tag = tagName, children = children)
                        } else {
                            val value = values.pop() ?: error(msg = "Missing value for Primitive tag!")
                            EmvTag.Primitive(tag = tagName, value = value)
                        }

                        if (isChild) {
                            children.peek()
                                ?.add(element = tag)
                                ?: error(msg = "Missing parent children list for tag!")
                        } else {
                            return@with tag
                        }
                    }

                    EventType.TEXT,
                    EventType.CDSECT -> values.push(element = text)

                    EventType.START_DOCUMENT,
                    EventType.ATTRIBUTE,
                    EventType.ENTITY_REF,
                    EventType.COMMENT,
                    EventType.DOCDECL,
                    EventType.IGNORABLE_WHITESPACE,
                    EventType.PROCESSING_INSTRUCTION -> { /* no-op */ }

                    EventType.END_DOCUMENT -> break
                }
                next()
            }

            error(msg = "Unable to parse EmvTag from input XML!")
        }

    override fun serializeNonXML(encoder: Encoder, value: EmvTag) {
        when (value) {
            is EmvTag.Primitive ->
                encoder.encodeSerializableValue(serializer = EmvTag.Primitive.serializer(), value = value)
            is EmvTag.Constructed ->
                encoder.encodeSerializableValue(serializer = EmvTag.Constructed.serializer(), value = value)
        }
    }

    override fun deserializeNonXML(decoder: Decoder): EmvTag =
        decoder.decodeSerializableValue(deserializer = EmvTagJsonDelegate.serializer()).asEmvTag()

    private fun error(msg: String): Nothing = throw SerializationException(msg)
}

private fun String.isConstructed(): Boolean = try {
    val firstByte = substring(0, 2).toInt(16)
    (firstByte and 0x20) != 0
} catch (_: Exception) { false }

Example usage:

val tag = EmvTag.Constructed(
        tag = "70",
        children = listOf(
            EmvTag.Primitive(tag = "9f14", value = "1234"),
            EmvTag.Primitive(tag = "9f15", value = "4321"),
            EmvTag.Constructed(
                tag = "77",
                children = listOf(EmvTag.Primitive(tag = "9f34", value = "9876")),
            ),
            EmvTag.Constructed(tag = "77", children = emptyList()),
        ),
    )

becomes

<EmvTag tag="70">
    <EmvTag tag="9f14">1234</EmvTag>
    <EmvTag tag="9f15">4321</EmvTag>
    <EmvTag tag="77">
        <EmvTag tag="9f34">9876</EmvTag>
    </EmvTag>
    <EmvTag tag="77" />
</EmvTag>

and

{
    "tag": "70",
    "children": [
        {
            "tag": "9f14",
            "value": "1234"
        },
        {
            "tag": "9f15",
            "value": "4321"
        },
        {
            "tag": "77",
            "children": [
                {
                    "tag": "9f34",
                    "value": "9876"
                }
            ]
        },
        {
            "tag": "77",
            "children": []
        }
    ]
}

Not the safest implementation considering all the XML events I ignored, but it works for my use-case since my XML inputs are always fairly simple.

[pdvrieze]

You want to probably handle ENTITY_REF as text (it is a subtype of it), but the events you are ignoring are actually safe to ignore (or don't actually occur: e.g. ATTRIBUTE).