Derive codec for parent sealed trait, given codecs for children

[kamilkloch]

Alternatively: provide means to chain codecs (e.g. as fallbackTo).

Imagine a following use-case: an API where codecs for the entire ADT cannot be derived automatically (e.g., due to inconsistent discriminator policy of the API). Concrete example: Kraken websocket API. Two top-level discriminators: method or channel, result containing an optional channel field (which only then is to be treated as a discriminator for a subtree of messages):

https://docs.kraken.com/api/docs/websocket-v2/book#subscribe-request

{
    "method": "subscribe",
    "result": {
        "channel": "book",
        "depth": 10,
        "snapshot": true,
        "symbol": "ALGO/USD"
    },
    "success": true,
    "time_in": "2023-10-06T17:35:55.219022Z",
    "time_out": "2023-10-06T17:35:55.219067Z"
}

https://docs.kraken.com/api/docs/websocket-v2/trade#subscribe-request

{
    "method": "subscribe",
    "result": {
        "channel": "trade",
        "snapshot": true,
        "symbol": "MATIC/USD"
    },
    "success": true,
    "time_in": "2023-09-25T09:21:10.428340Z",
    "time_out": "2023-09-25T09:21:10.428375Z"
}

https://docs.kraken.com/api/docs/websocket-v2/add_order#response

{
    "method": "add_order",
    "req_id": 123456789,
    "result": {
        "order_id": "AA5JGQ-SBMRC-SCJ7J7",
        "order_userref": 100054
    },
    "success": true,
    "time_in": "2023-09-21T14:15:07.197274Z",
    "time_out": "2023-09-21T14:15:07.205301Z"
}

https://docs.kraken.com/api/docs/websocket-v2/trade#snapshot-and-update-response

{
    "channel": "trade",
    "type": "snapshot",
    "data": [
        {
            "symbol": "MATIC/USD",
            "side": "buy",
            "price": 0.5147,
            "qty": 6423.46326,
            "ord_type": "limit",
            "trade_id": 4665846,
            "timestamp": "2023-09-25T07:48:36.925533Z"
        },
        {
            "symbol": "MATIC/USD",
            "side": "buy",
            "price": 0.5147,
            "qty": 1136.19677815,
            "ord_type": "limit",
            "trade_id": 4665847,
            "timestamp": "2023-09-25T07:49:36.925603Z"
        }
    ]
}

In short, a standard automatic top-down approach of deriving codecs (start with a top sealed trait, use discriminators at each nested level to find a first match of a subtype) fails short here.

A working solution: derive codecs for each message type separately (via JsonCodecMaker.make), then manually define codec for the parent sealed trait by combining codecs of the children with the firstOf policy. (see https://circe.github.io/circe/codecs/adt.html).

This is approach is currently difficult to implement, as the parent codec must actually dwell into the internals of child messages (see e948fb9, relates to #1266). Ideally, we would like to be able to combine child codecs as black boxes, without manually applying multiple low-level discriminating steps in the parent codec.

[plokhotnyuk]

Hi, Kamil!

I cannot understand how it differs from the #1266 case.

Could you please prepare a code snippet on Scastie site with your definitions and related JSON inputs?

[kamilkloch]

I cannot understand how it differs from the #1266 case.

The proposed solution for decoder (e948fb9) goes into the internals of the leaves. Assume the leaf codecs are black boxes, how should one then implement the decoder?

[plokhotnyuk]

After successful scanning for the top-level discriminator with in.skipToKey("result") you can go into the next JSON object level by in.isNextToken('{') call with subsequent scanning for the nested discriminator by in.skipToKey("channel").

[kamilkloch]

Yes, but it is quite low-level, asserts knowledge of jsoniter internals (which, arguably, should not be required) and most importantly, unnecessarily spill the internals of leaf codecs (discriminators) into the parent codec.

[plokhotnyuk]

I'd love to create a custom codec tailored to your needs! To get started, could you please share the definitions of your data structures paired with corresponding JSON input?

Crafting custom codecs is similar to using Circe's combinators. We'll leverage the library's internals to tackle unique cases that macros can't handle. Unlike Circe, we won't have an Abstract Syntax Tree (AST) to work with, so we'll operate directly with tokens. This approach is key to achieving outstanding performance and scalability for your codecs.

[kamilkloch]

Hello Andriy,

Thank you so much. I will try to write custom codecs myself, and perhaps consult with you for pitfalls and mistakes that definitely will be there ;) Thank you!

[kamilkloch]

I'd love to create a custom codec tailored to your needs! To get started, could you please share the definitions of your data structures paired with corresponding JSON input?

Hello Andriy, I am stuck ;) An ADT with a top discriminator channel, but for channel value balances forking into two different messages, depending on the value of field type:

• if type == snaphost, then https://docs.kraken.com/api/docs/websocket-v2/balances#snapshot-response
• if type == update, then https://docs.kraken.com/api/docs/websocket-v2/balances#update-response

UPDATE: I have some idea, will try to implement it later today and will come back.

[plokhotnyuk]

Hi, Kamil! I'm happy to hear that you going to find a solution using jsoniter-scala.

I can only guess without data structure definitions and corresponding JSON samples.

Am I understand right that the same web-socket connection is used to receive different types of data from different channels and the main challenge is to select a proper instance of the codec?

For generic case we can use 2 different data structures and related codecs for the same input: 1st one will be used for parsing of all discriminators, skipping all other fields; 2nd one will use parsed discriminators to re-parse a message by a proper codec that corresponds to the valid combination of discriminator values.

[kamilkloch]

Hello Andriy,

Yes, I will try this approach and share the code. Thank you!

[kamilkloch]

Hello Andriy. Below a custom codec for KrakenWsChannelMessage, where the discriminator field is channel, unless channel == balances - then the discriminator field is type. I used 2 sub-hierarchies (ChannelDiscriminator, TypeDiscriminator) for these cases. Thank you in advance for any hints!

sealed trait KrakenWsChannelMessage

object KrakenWsChannelMessage {
  sealed trait ChannelDiscriminator extends KrakenWsChannelMessage

  case object Heartbeat extends ChannelDiscriminator

  // ... plus other instances of ChannelDiscriminator

  sealed trait TypeDiscriminator extends KrakenWsChannelMessage

  case class Snapshot(channel: "balances", data: List[AssetObject]) extends TypeDiscriminator

  case class Update(channel: "balances", data: List[LedgerTransactionObject]) extends TypeDiscriminator

  private val channelDiscriminatorCodec: JsonValueCodec[ChannelDiscriminator] =
    JsonCodecMaker.make(codecMakerConfig.withDiscriminatorFieldName(Some("channel")))

  private val typeDiscriminatorCodec: JsonValueCodec[TypeDiscriminator] =
JsonCodecMaker.make(codecMakerConfig.withDiscriminatorFieldName(Some("type")).withRequireDiscriminatorFirst(false))

  val codec: JsonValueCodec[KrakenWsChannelMessage] = new JsonValueCodec[KrakenWsChannelMessage] {
    override def decodeValue(in: JsonReader, default: KrakenWsChannelMessage): KrakenWsChannelMessage = {
      in.setMark()
      if (in.isNextToken('{')) {
        if (in.skipToKey("channel")) {
          val channel = in.readString(null)
          in.rollbackToMark()
          if (channel == "balances")
            typeDiscriminatorCodec.decodeValue(in, typeDiscriminatorCodec.nullValue)
          else
            channelDiscriminatorCodec.decodeValue(in, channelDiscriminatorCodec.nullValue)
        } else {
          in.resetMark()
          in.requiredFieldError("channel")
        }
      } else {
        in.resetMark()
        in.readNullOrTokenError(default, '{')
      }
    }

    override def encodeValue(x: KrakenWsChannelMessage, out: JsonWriter): Unit = x match {
      case x: ChannelDiscriminator => channelDiscriminatorCodec.encodeValue(x, out)
      case x: TypeDiscriminator => typeDiscriminatorCodec.encodeValue(x, out)
    }

    override def nullValue: KrakenWsChannelMessage = null.asInstanceOf[KrakenWsChannelMessage]
  }
}