Unable to deserialize a previously serialized object - Pending Callbacks and Reference Resolvers

[mressler]

What is the expected and actual behavior?

Context.ts:65 is assuming that when context.pendingCallbacks == context.pendingRefsCount that all pending callbacks are pending reference resolvers. This is not the case.

It is possible for pendingCallbacks to be equivalent to pendingRefsCount when there are many more unresolved forward references. Here is a test case that fails:

it("(de)serializes class hierarchy with many forward references", () => {
        class EventState {
            @serializable(identifier())
            id: string

            @serializable
            name: string
        }

        class Event {
            @serializable(identifier())
            id: string

            @serializable
            name: string

            @serializable(reference(EventState))
            startingState: EventState

            @serializable(reference(EventState))
            endingState: EventState

            @serializable(list(reference(EventState)))
            relatedEvents: Event[]
        }

        class EventListing {
            @serializable(identifier())
            id: string

            @serializable(list(object(Event)))
            events: Event[]

            @serializable(list(object(EventState)))
            eventStates: EventState[]
        }

        const state1 = new EventState()
        state1.id = "state1"
        state1.name = "State 1"

        const state2 = new EventState()
        state2.id = "state2"
        state2.name = "State 2"

        const event1 = new Event()
        event1.id = "event1"
        event1.name = "Event 1"
        event1.startingState = state1
        event1.endingState = state2
        event1.relatedEvents = []

        const event2 = new Event()
        event2.id = "event2"
        event2.name = "Related Event 2"
        event1.relatedEvents.push(event2)

        const event3 = new Event()
        event3.id = "event3"
        event3.name = "Related Event 3"
        event1.relatedEvents.push(event3)

        const event4 = new Event()
        event4.id = "event4"
        event4.name = "Related Event 4"
        event1.relatedEvents.push(event4)

        const eventListing = new EventListing()
        eventListing.id = "listing1"
        eventListing.events = [event1, event2, event3, event4]
        eventListing.eventStates = [state1, state2]

        const serialized = serialize(eventListing)
        console.log("Serialized event listing:", JSON.stringify(serialized, null, 2))

        expect(serialized.id).toBe(eventListing.id)
        expect(serialized.events.length).toBe(4)
        expect(serialized.eventStates.length).toBe(2)
        expect(serialized.events[0].id).toBe(event1.id)
        expect(serialized.events[0].startingState).toBe(state1.id)
        expect(serialized.events[0].endingState).toBe(state2.id)
        expect(serialized.events[0].relatedEvents.length).toBe(3)
        expect(serialized.events[0].relatedEvents[0]).toBe(event2.id)
        expect(serialized.events[0].relatedEvents[1]).toBe(event3.id)
        expect(serialized.events[0].relatedEvents[2]).toBe(event4.id)

        const deserEventListing = deserialize(EventListing, serialized)
        expect(deserEventListing.id).toBe(eventListing.id)
        expect(deserEventListing.events.length).toBe(4)
        expect(deserEventListing.eventStates.length).toBe(2)
        expect(deserEventListing.events[0].id).toBe(event1.id)
        expect(deserEventListing.events[0].startingState.id).toBe(state1.id)
        expect(deserEventListing.events[0].endingState.id).toBe(state2.id)
        expect(deserEventListing.events[0].relatedEvents.length).toBe(3)
        expect(deserEventListing.events[1].id).toBe(event2.id)
        expect(deserEventListing.events[1].startingState.id).toBe(state1.id)
        expect(deserEventListing.events[1].endingState.id).toBe(state2.id)
    })

This produces the following output:

Serialized event listing: {
        "id": "listing1",
        "events": [
          {
            "id": "event1",
            "name": "Event 1",
            "startingState": "state1",
            "endingState": "state2",
            "relatedEvents": [
              "event2",
              "event3",
              "event4"
            ]
          },
          {
            "id": "event2",
            "name": "Related Event 2",
            "startingState": null,
            "endingState": null
          },
          {
            "id": "event3",
            "name": "Related Event 3",
            "startingState": null,
            "endingState": null
          },
          {
            "id": "event4",
            "name": "Related Event 4",
            "startingState": null,
            "endingState": null
          }
        ],
        "eventStates": [
          {
            "id": "state1",
            "name": "State 1"
          },
          {
            "id": "state2",
            "name": "State 2"
          }
        ]
      }

      at Object.<anonymous> (test/typescript/ts.test.ts:1059:17)

  ● @subSchema › (de)serialize class hierarchy with many forward references

    Error: Unresolvable references in json: "state1", "state2", "event2", "event3", "event4"

    

      at Context.GUARDED_NOOP [as onReadyCb] (src/utils/utils.ts:2067:11)
      at src/core/Context.ts:1767:20
      at src/utils/utils.ts:2093:14
      at onAfterDeserialize (src/core/deserialize.ts:2802:5)
      at src/core/deserialize.ts:2845:45
      at src/types/identifier.ts:527:9
      at Object.deserializer (src/types/primitive.ts:322:19)
      at Object.deserializer (src/types/identifier.ts:509:41)
      at deserializeProp (src/core/deserialize.ts:2837:13)
      at callbackDeserialize (src/core/deserialize.ts:2922:9)
      at onBeforeDeserialize (src/core/deserialize.ts:2777:5)
      at deserializePropsWithSchema (src/core/deserialize.ts:2931:37)
      at deserializeObjectWithSchema (src/core/deserialize.ts:2750:3)
      at Object.deserializer (src/types/object.ts:825:65)
      at callbackBefore (src/types/list.ts:1068:24)
      at onBeforeDeserialize (src/core/deserialize.ts:2777:5)
      at processItem (src/types/list.ts:1094:47)
      at src/utils/utils.ts:2138:5
          at Array.forEach (<anonymous>)
      at parallel (src/utils/utils.ts:2134:6)
      at Object.deserializer (src/types/list.ts:1098:28)
      at deserializeProp (src/core/deserialize.ts:2837:13)
      at callbackDeserialize (src/core/deserialize.ts:2922:9)
      at onBeforeDeserialize (src/core/deserialize.ts:2777:5)
      at deserializePropsWithSchema (src/core/deserialize.ts:2931:37)
      at deserializeObjectWithSchema (src/core/deserialize.ts:2750:3)
      at deserialize (src/core/deserialize.ts:2703:12)
      at Object.<anonymous> (test/typescript/ts.test.ts:1070:63)

I'm not sure what that line is guarding for. It seems like we should be able to decrement pendingCallbacks and only call the completion handler when pendingCallbacks == 0?

I'm playing around with this and will hopefully provide a PR. Happy to take direction.

[mressler]

The code in question is this:

                fn(value!);
                if (--this.pendingCallbacks === this.pendingRefsCount) {
                    if (this.pendingRefsCount > 0) {
                        // all pending callbacks are pending reference resolvers. not good.
                        this.onReadyCb(
                            new Error(
                                `Unresolvable references in json: "${Object.keys(this.pendingRefs)
                                    .filter((uuid) => this.pendingRefs[uuid].length > 0)
                                    .join(
                                        // prettier-ignore
                                        "\", \""
                                    )}"`
                            )
                        );
                        Context.rootContextCache.delete(this);
                    } else {
                        this.onReadyCb(null, this.target);
                        Context.rootContextCache.delete(this);
                    }
                }

fn(value!); is the completion - everything else is bookkeeping. Obviously we need to decrement the pendingCallbacks and it seems smart to call the onReadyCb to when we have 0 pending callbacks because then we should be done. But I don't understand the comment there - why is it the case that pending callbacks ARE pending reference resolvers?

Instead of running this check here, I'm planning on wrapping the onReadyCb with the check to determine if there are pending reference callbacks. Then we can guarantee that we're about to return the deserialized value to the user and can do the right thing there.