Graph backend phase 1

[compulim]

Changelog Entry

Added

• New JSON-LD graph backend, by @compulim in PR #5622

Description

This is phase 1 of the work. More phases are coming soon.

We are introducing a new JSON-LD graph backend. This new backend is paving the way towards replacing Redux as the single source of truth. And in future, next generation chat adapter will emit nodes and edges, instead of activities.

The graph backend use JSON-LD to ensure data are linked in a standardized manner. The schema will be largely backed by Schema.org along with some custom fields to support backward compatibility.

In this pull request:

• (Existing) Chat adapter insert activities into Redux store, as single source of truth
  • 🆕 channelData['webchat:internal:id']: string keeps the (semi-)permanent ID, similar to <ActivityKeyer> and is intended to replace <ActivityKeyer> in the future
  • 🆕 channelData['webchat:internal:position']: number keeps the positional index of the activity in the chat history
• 🆕 Activities from Redux store transform into JSON-LD node and inserted into graph
  • Activity (as JS object) is kept as-is under custom property named urn:microsoft:webchat:direct-line-activity:raw-json
• 🆕 On graph change, query all Message nodes and sort them into an array orderedActivities: readonly WebChatActivity[]
• 🆕 useActivities() will read from the new orderedActivities, instead of from Redux store state
  • Assertion is added to make sure result from orderedActivities is 100% same as Redux store state
  • The assertion will be removed once Redux store is no longer the single source of truth

What is (obviously) missing from this phase:

• Web Chat still use Direct Line Activity for everything, instead of JSON-LD nodes
• activity.attachments[] and activity.entities[] are not serialized to the graph yet
• No new hooks to access JSON-LD nodes
• Message schema is not formalized yet
• node.sender property is of type Audience and this is not coerrect, should be more targeted thing, such as Person or Organization
• No way to dump graph and incoming activities
• activity:raw-json should be JSON literals (all values must be serializable as JSON)
• Replace <ActivityKeyer> with the new permanent ID
• How to modify polymiddleware to support node while keeping backward compatibility

Design

JSON-LD Opinions: halfway between styles

JSON-LD is great for storing node and edges. However, to effectively access and retrieve data from the graph for our use, the data is shaped with opinions, around a few principles:

• Simplifying downstream logics
  • Must have @id: every node in the graph must be identifiable
  • Uniform getter/setter: every property value is an array, except @context and @id
  • Unique typing: node reference must be { "@id": string } to reduce confusion with plain string
  • Support multiple types: every @type must be an array of string
  • Reduce confusion: property value with empty array and null is removed
    • [] and null are same as if the property is removed
  • Flattened: property values must be non-null literals, node reference, or JSON literals
    • Any array containing null is not supported and will throw unless it is JSON literal, as it is likely a bug in code
  • JSON literals will have boxing kept: { '@type': '@json', '@value': JSONValue }
    • @value could be null, if unwrapped, will be confusing as we removed nulls
  • Do not handle full JSON-LD spec: @context is an opaque string and its schema is not honored
  • Node reference only has @id and it should not contain @type
  • Reduce confusion: node reference must not appear at the root of the flattened graph, they are semantically empty
• Auto-linking for Schema.org: hasPart and isPartOf are auto-inversed
• Keep its root: every node is compliant to JSON-LD, understood by standard parsers
• Debuggability: must have at least one @type

The end result is still 100% legit JSON-LD. However, they are not 100% styled as flattened nor compact. In other words, the final data shape is not absolutely styled, but halfway between each style to better fit our scenario.

Refactoring insertion sort algorithm

1. By part grouping ID/position
2. By livestreaming session ID/position
3. By sequence ID coalesced with Unix epoch timestamp
   • If sequence ID is unset, use Unix epoch timestamp instead
   • When inserting, ignore existing activities where they have no sequence ID nor timestamp
4. Otherwise, append to the chat history

This refactoring allows us to have a new channelData['webchat:internal:position']: number property to keep the positional index of the activity. We will put this number into Message.position property.

Specific Changes

-

• I have added tests and executed them locally
• I have updated CHANGELOG.md
• I have updated documentation

Review Checklist

This section is for contributors to review your work.

• Accessibility reviewed (tab order, content readability, alt text, color contrast)
• Browser and platform compatibilities reviewed
• CSS styles reviewed (minimal rules, no z-index)
• Documents reviewed (docs, samples, live demo)
• Internationalization reviewed (strings, unit formatting)
• package.json and package-lock.json reviewed
• Security reviewed (no data URIs, check for nonce leak)
• Tests reviewed (coverage, legitimacy)

[OEvgeny]

Minor nits and questions

[Copilot]

@OEvgeny I've opened a new pull request, #5631, to work on those changes. Once the pull request is ready, I'll request review from you.