Proposal: Improve mapping

[nikgraf]

Currently we use the type IDs and property IDs to map between the local Schema and the public Graph. For private spaces we use the names of the actual schema e.g.

export class Todo extends Entity.Class<Todo>('Todo')({
  name: Type.String,
  completed: Type.Boolean,
}) {}

in Automerge stores:

{
  entities: [
    {
       name: 'call CEO',
       completed: true,
       _types: ['Todo'] 
    }
  ]
}

This is problematic since it doesn't allow for interoperability in private spaces since all of them would need to implement the same Schema names. Instead we want:

{
  entities: [
    {
      'a126ca53-0c8e-48d5-b888-82c734c38935': 'call CEO',
      '1765d387-6c28-4baa-9ed8-061704fa010d': true,
      _type: ['ab700a03-ae75-4fdc-8806-cb8445dcd9b5']
    }
  ]
}

This means every Type schould have at least one valid type ID and for ever property a valid id.

Issues with the current mapping file structure

• We don't have any validation of Schema to Mapping
• The mapping file is globally defined and to be passed into the AppProvider. (No code-splitting possible of the mapping, not a good architecture for large applications)
• We have several places where we map the schema/mapping and if doesn't fit we have to bail out with errors during runtime. That's quite ugly. Ideally we can throw an error right in the beginning when starting the programm.

Proposal

We inline the mapping IDs directly into the schema. This way we can have static

export class User extends Entity.Class<User>('User', '0929ef5e-746b-4cce-aaeb-2a7e101c2e55')({
  name: Type.String('a126ca53-0c8e-48d5-b888-82c734c38935'),
}) {}

export class Todo extends Entity.Class<Todo>('Todo', 'ab700a03-ae75-4fdc-8806-cb8445dcd9b5')({
  name: Type.String('a126ca53-0c8e-48d5-b888-82c734c38935'),
  completed: Type.Boolean('1765d387-6c28-4baa-9ed8-061704fa010d'),
  assignees: Type.Relation(User, '1203064e-9741-4235-89d4-97f4b22eddfb'),
}) {}

This means we also could provided default types like we provide System IDs on grc-20-ts e.g.

import { SystemTypes, SystemIds } = "@graphprotocol/hypergraph"

export class User extends Entity.Class<User>('User', '0929ef5e-746b-4cce-aaeb-2a7e101c2e55')({
  name: SystemTypes.name, // more convenient than Type.String('a126ca53-0c8e-48d5-b888-82c734c38935')
}) {}

export class Todo extends Entity.Class<Todo>('Todo', 'ab700a03-ae75-4fdc-8806-cb8445dcd9b5')({
  name: SystemTypes.name,
  completed: SystemTypes.completed,
  assignees: Type.Relation(User, SystemIds.assignees),
}) {}

Benefits

• If you don't provide a string TS will error!
• We can check the validity of the mapping right at the beginning when the program starts compared to when you actually start a query (better for DX)

Note: Under the hood we would attach the ID as annotation to Effect. This would be very nice to access every time we need to convert data between local version to be used by devs to the public graph/automerge storage version.

  name: Type.String.pipe(Schema.annotations({
    [Hypergraph.Symbol]: "'f60585af-71b6-4674-9a26-b74ca6c1cceb'"
  })),

Alternative design

Why I would not like to go with it:

• Lot's of unnecessary code duplication (the whole structure is duplicated)
• This could work, but is much harder to make type safe and afaik we can't use Effect Schema annotations

export class User extends Entity.Class<User>('User', '0929ef5e-746b-4cce-aaeb-2a7e101c2e55')({
  name: Type.String,
}) {
  mapping = {
    typeIds: ['0929ef5e-746b-4cce-aaeb-2a7e101c2e55'],
    properties: {
      name: Id('a126ca53-0c8e-48d5-b888-82c734c38935'),
    },
  }
}

export class Todo extends Entity.Class<Todo>('Todo', 'ab700a03-ae75-4fdc-8806-cb8445dcd9b5')({
  name: Type.String,
  completed: Type.Boolean,
  assignees: Type.Relation(User),
}) {
  mapping = {
    typeIds: [Id('0929ef5e-746b-4cce-aaeb-2a7e101c2e55')],
    properties: {
      name: Id('a126ca53-0c8e-48d5-b888-82c734c38935'),
      completed: Id('1765d387-6c28-4baa-9ed8-061704fa010d'),
    },
    relations: {
       assignees: ['1203064e-9741-4235-89d4-97f4b22eddfb']
    }
  }
}

Open Question

• Can I put a 'TODO' or other placeholder there? If yes, we shouldn't could still do UUID validation, but show a warning in the console.

[baiirun]

I like that this makes the mapping much more type safe. I also like that we can combine multiple concepts (the schema + the mapping) into just one thing. My only concern is if people will be turned away by the "magic" strings everywhere in their schema. I suppose they could just store them in a variable like you did for the system properties.

In a similar vein, have you thought about how schema migrations work with Hypergraph?

[nikgraf]

@baiirun What do you mean by? Do you maybe have a concrete example?

I suppose they could just store them in a variable like you did for the system properties.

I thought a bit about schema migrations in the past, but this was all based on different assumptions. No clear idea how to do this with our public graph data model

[baiirun]

What do you mean by?

const BananaProperty = Type.String("banana-id");
const UserId = "0929ef5e-746b-4cce-aaeb-2a7e101c2e55"

export class User extends Entity.Class<User>('User', UserId)({
  banana: BananaProperty,
}) {}

[nikgraf]

ahhh, so just moving code so it's easier to read. Works for me! That said would this complicate Typesync @cmwhited or do you handle such cases anyway already?

[baiirun]

IMO let's just do the IDs inline for now like in your proposal. If developers find it annoying we can find a fix for it later.

[cmwhited]

export class User extends Entity.Class<User>('User', '0929ef5e-746b-4cce-aaeb-2a7e101c2e55')({
  name: Type.String('a126ca53-0c8e-48d5-b888-82c734c38935'),
}) {}

export class Todo extends Entity.Class<Todo>('Todo', 'ab700a03-ae75-4fdc-8806-cb8445dcd9b5')({
  name: Type.String('a126ca53-0c8e-48d5-b888-82c734c38935'),
  completed: Type.Boolean('1765d387-6c28-4baa-9ed8-061704fa010d'),
  assignees: Type.Relation(User, '1203064e-9741-4235-89d4-97f4b22eddfb'),
}) {}

From an engineering perspective, I definitely prefer this approach, mostly because of the built-in type-safety it gives us. So this gets my vote.

My only concern is if people will be turned away by the "magic" strings everywhere in their schema

I do echo this concern.

Related, it has me thinking about the development workflow for building a schema, whether through the schema.ts file, or through typesync.

Currently:

1. user creates a schema.ts file, imports from @graphprotocol/hypergraph
2. they start building a schema:

import { Entity, Type } from '@graphprotocol/hypergraph';

export class User extends Entity.Class<User>('User')({
  name: Type.String,
}) {}

3. they generate a mapping for this schema that generates the GRC-20 id values.

With this new approach, a user would need to generate the GRC-20 id in-line with building their schema. Or, how do you see this happening? Cause it feels to me that this would require they start building the GRC-20 createType/Property before building the schema?? Or would hypergraph handle that?? cc @nikgraf

[nikgraf]

@cmwhited I was thinking along these lines:

If they manually create the code. This would show TS errors and would be invalid due the missing strings:

import { Entity, Type } from '@graphprotocol/hypergraph';

export class User extends Entity.Class<User>('User')({
  name: Type.String,
}) {}

They could do this one:

import { Entity, Type } from '@graphprotocol/hypergraph';

export class User extends Entity.Class<User>('User', 'todo-user-type-id')({
  name: Type.String('todo-name-id'),
}) {}

Then they could go into Typesync and update the fields. Maybe Typesync could even be smart about it and show: we here you have an invalid ID and suggest the one the makes the most sense.

btw the todo string would still work and could even use if for local development. The data in Automerge would be mapped incorrectly (so you would have invalid entries).

Note: I'm very unhappy about all of the ID mapping. Trying to find something that solves the Automerge storage problem and has the least DX impact. If you have any ideas! Please let me know :)

[cmwhited]

import { Entity, Type } from '@graphprotocol/hypergraph';

export class User extends Entity.Class<User>('User', 'todo-user-type-id')({
  name: Type.String('todo-name-id'),
}) {}

Hmmmm. This feels confusing to me. Like would require a lot of documentation, both in the code and externally, to make sure users understand they can do this.
I wonder if maybe we have 2 modes:

1. local only/unpublished

import { Entity, Type } from '@graphprotocol/hypergraph';

export class User extends Entity.Class<User>('User')({
  name: Type.String,
}) {}

2. published

import { Entity, Type } from '@graphprotocol/hypergraph';

export class User extends Entity.Class<User>('User', '0929ef5e-746b-4cce-aaeb-2a7e101c2e55')({
  name: Type.String('a126ca53-0c8e-48d5-b888-82c734c38935'),
}) {}

And could even build in to the publishing flow through typesync that it updates the schema.ts and adds the correct ids.

Thoughts?

[nikgraf]

yeah agreed on this would be confusing. We could make your version work. Would be less typesafe but better DX.

looking at it from a new perspective today and thinking: this looks scary

import { Entity, Type } from '@graphprotocol/hypergraph';

export class User extends Entity.Class<User>('User', '0929ef5e-746b-4cce-aaeb-2a7e101c2e55')({
  name: Type.String('a126ca53-0c8e-48d5-b888-82c734c38935'),
}) {}

will sleep over it, in case you have more ideas! please share