Json Inference

[BarryThePenguin]

Hi, I'm looking for ways to infer a schema from some sample Json documents.

Something like https://app.quicktype.io is able to infer a Json Schema from multiple documents, while also detecting enums and objects with similar keys. But they don't expose any underlying AST, so you're limited to the output formats they provide.

I've played around with @traversable/json and @traversable/json-schema with Json.fold<() => JsonSchema>.
I've also used the resulting JsonSchema and @traversable/valibot with JsonSchema.fold<vx.AnyValibotSchema>.
I'm having a few issues handling JsonSchema.Ref correctly, but it's probably just an issue on my part 😅

Ideally I'd like to go from @traversable/json -> @traversable/schema -> @traversable/valibot rather than using JsonSchema as an in-between step.

Do you have any advice as to how I might do this?

[ahrjarrett]

Hey! Not sure if this is what you're looking for, the @traversable/zod package has a function called fromJson that sounds similar.

Here's a small POC of how you could do something similar for Valibot.

If you're looking for something else, could you give me an example JSON input, and the schema you're hoping to get out the other end?

[ahrjarrett]

I took a closer look at quicktype, and now I think I better understand. It sounds like the intermediate representation is specifically what you're looking for.

If so, I'd definitely be interested in helping figure out how to support your use case.

The part that makes me think I don't fully understand your use case is how refs come into play, if your source is JSON. That seems to imply that you might be working with multiple JSON documents, and making a separate pass to perform inference.

I'd need to understand a bit more about how that step works to be able to help!

[BarryThePenguin]

For a bit more context, I'm looking at sample JSON documents from undocumented APIs and using them to infer a schema.

One example could be a number of weather observations

{
  "data": {
    "temperature": 10,
    "wind_speed": 30,
    "wind_direction": "N"
  }
}

{
  "data": {
    "temperature": 15,
    "wind_speed": 0,
    "wind_direction": null
  }
}

{
  "data": {
    "temperature": 20,
    "wind_speed": 15,
    "wind_direction": "W"
  }
}

Another example could be a list of tasks in a project

{
  "data": [
    {
      "id": 123456,
      "assignee_id": 98765,
      "content": "A task",
      "priority": "low"
    },
    {
      "id": 234567,
      "assignee_id": null,
      "content": "Another task",
      "priority": "high"
    },
    {
      "id": 345678,
      "assignee_id": 98765,
      "content": "Yet another task",
      "priority": "medium"
    }
  ]
}

Without having a large number of sample JSON documents to infer from, I'd be happy to make a few assumptions, as long as I could verify those assumptions somehow. Some of these assumptions might include

• Specifying a path to a reference could setup an object, or objects in an array, to provide a definition for that reference

Json.toJsonSchema(json, {
  refs: { Task: ["data"] },
});

{
  "$defs": {
    "Task": {
      "type": "object",
      "properties": "..."
    }
  },
  "properties": {
    "data": {
      "type": "array",
      "items": {
        "$ref": "#/$defs/Task"
      }
    }
  }
}

• Specifying a path to an enum could replace any values with a reference to that enum definition

Json.toJsonSchema(json, {
  enums: {
    WindDirection: {
      path: ["data", "wind_direction"],
      values: ["N", "W", "S", "E"],
    },
  },
});

{
  "$defs": {
    "WindDirection": {
      "enum": ["N", "W", "S", "E"]
    }
  },
  "properties": {
    "data": {
      "type": "object",
      "properties": {
        "wind_direction": {
          "$ref": "#/$defs/WindDirection"
        }
      }
    }
  }
}

[BarryThePenguin]

If so, I'd definitely be interested in helping figure out how to support your use case.

I could be overthinking this 😅 an alternative would be to write any schema/types by hand and adjust it as I find new data that doesn't fit my initial assumptions. Being able to infer the schema and generate types would be nice though!

The part that makes me think I don't fully understand your use case is how refs come into play, if your source is JSON. That seems to imply that you might be working with multiple JSON documents, and making a separate pass to perform inference.

Yes, I think that sounds right. Given multiple JSON documents I'd make seperate passes over them. Possibly similar to what JsonSchema.fold does to resolve refs.

schema/packages/json-schema-types/src/functor.ts

Lines 139 to 145 in 4868df1

	export const fold = (<T>(g: (src: JsonSchema.F<T>, ix: Index<T>, x: JsonSchema) => T) => (x: JsonSchema.F<T>, ix?: Index) => {
	const refs = fn.map(resolveRefs(x), (schema) => () => fold_(g)(schema, ix))
	return {
	refs,
	result: fold_(g)(x, { ...defaultIndex, refs }),
	}
	}) as Fold

But in this instance, the refs would need to be amended when traversing multiple samples