An acceptable way to define schemas in Gleam

[massivefermion]

As you may or may not know, I'm the developer of the gleam_mongo package and you can see in the roadmap that I plan to add more features to it so that someday people can actually use it in production. But there is one feature that comes to my mind from time to time that I can't bring myself to put in the roadmap because of one particular problem that I can't seem to find a good way to solve and that feature is supporting mongodb's schema validation.
I've also thought about writing a graphql library for Gleam, but I had to put that thought aside because of the same problem I mentioned above and that problem is:

• How can developers define their graphql/mongodb schema?

I'm pretty sure anybody that reads this, instantly thinks of a way to do this in Gleam that may actually look satisfactory to them but since Gleam isn't stable yet, I think it makes sense to not be satisfied so easily and actually look for a better way. So let me go through my thought process to show why I don't think there is a good solution for the mentioned problem in Gleam yet!

The obvious solution that comes to mind, is to have a function that developers can use to define their schemas. This function will receive e.g. a document of type List(#(String, String)). I'm also sure you can come up with a type that allows for nested schemas. But my problem with this approach is that inevitably the developer needs to define some Gleam types equivalent with the schemas they define and that means there are now multiple sources of truths for the entities that the developer needs to define to implement their business logic and that seems like bad DX to me!

Another solution is to use non-Gleam code to define the schemas as a single source of truth and then auto-generate Gleam types from them. For graphql you can use gql files and for mongodb you can use json schemas inside json files. Then you can run a task that goes through those schemas and creates some Gleam files that contain the equivalent Gleam types. This has the advantage that we now have a single source of truth for the business entities we want to define. But there are several disadvantages. First: What if we want to use mongodb schema validation and graphql in the same project? Then we'll have the same problem of multiple sources of truths. Second: It would be ideal if we could let Gleam developers use Gleam to solve their problem. Third: Gleam currently does not have a standard way of running predefined tasks (But I think implementing one is a milestone, so I mentioned it as the third problem).

The last solution that comes to my mind, and I really think is the ideal solution, is that the developer only needs to define Gleam types and those types can be used to generate mongodb and graphql schemas. I've actually tried to accomplish this. Of course the types themselves will not be there when you're running your code but you can inspect values created using those types using the dynamic module. But the problem is that those values don't retain the field names in the custom type so I can't actually use them to define a schema. Another problem is that fields in custom types can't have default values, so you have to give each of them a dummy value to create the entity that you want to inspect, which means you can't actually use this solution in the general case when you don't have an idea what the fields and their types are.

As I said the last solution seems to me to be the ideal solution to this problem but you can't actually implement it using Gleam. I understand that retaining full type information at runtime may seem wasteful given that this may be the only problem that it may solve, but I just want to say that if this is actually supported, people will probably find more use cases for it. There is also the option to just retain type information for a few selected types. So maybe you can have e.g. a firm keyword like this:

pub firm type User {
  User(username: String, age: Int)
}

And maybe if you define some types as firm, the Gleam compiler creates some kind of a data structure that contains all the information about all those firm types that can somehow be accessed at runtime.

These are just some primitive thoughts of someone who doesn't have a lot of practical knowledge about implementing a compiler so I don't think I can actually formulate a concrete proposal, I'm not even sure if this actually makes sense. But hopefully we can have a fruitful discussion about this issue.

EDIT: I was searching for something similar for typescript/javascript and I found this which seems interesting!

[inoas]

What would help me a lot is being able to reuse variant type values without wrapping them so I can define them somewhere and import them.

That being said you can create a field that holds the name so you can access that at runtime, pretty much like Elixir Structs hold a name AFAIR.

Maybe it would be helpful to have compile time constants #1501 and one could be __TYPE__ or __VARIANT__ or something which at compile time inserts values somehow so authors can transfer some compile time info statically into the runtime without having to duplicate naming (and keep it consistent)?

[lpil]

One existing example of the creation of a schema is the dynamic.Decoder type, which is constructed using a functional DSL. Understanding this DSL is a bit of a barrier to entry!

And maybe if you define some types as firm, the Gleam compiler creates some kind of a data structure that contains all the information about all those firm types that can somehow be accessed at runtime.

Could you share some ideas and possibly pseudocode on how you might use this feature? Thanks!

[lpil]

Great, thank you. In this case the programmer would still need to write the boilerplate for the insertion code, is that right?

[massivefermion]

Actually I've thought about that too. I think it would be great if we could generate functions that accept the types written by the developer and just create the document without the need for the developer to use the bson library. It has to be more like a task thing, because you'd run it once(more like every time you change your types) and it writes the boilerplate to a file and then you can use it. But without __TYPES__, implementing it requires actually parsing the types and getting a typed AST and using that to generate the boilerplate.

I've actually thought about forking the Gleam compiler to make a separate executable that uses the typed AST generated by the compiler to does all that but requiring the developer to deal with another executable just to work with mongodb seems absurd to me. but with this feature, I think the insertion boilerplate can be easily generated too.

[lpil]

Did you have particular ideas for this code generation? There are lots of options here: type classes, macros, reflection, external code generators, parse transforms, etc

[massivefermion]

Well, I know macros are not supported in Gleam(and I don't think adding them to Gleam just because of this is a good idea), so I just didn't think about them, and to be honest I don't exactly know what you mean by type classes and parse transforms. And if I understand correctly, reflection means being able to access(or maybe even alter) the types at runtime which is exactly what __TYPES__ provides.
But before that I thought if I could somehow intercept Gleam's compiler at the stage that it has the typed AST, I could go through it and find the types that the developer wants to use for insertion into collections and generate functions for them like below:

pub fn insert_book(book_collection: Collection, book: Book) {
  let bson_book = types.Document([
    #("title", types.Str(book.title)),
    #("pages", types.Integer(book.pages)),
    #("author", types.Str(book.author)),
  ])
  
  mongo.insert_one(book_collection, bson_book)
}

That's usually how I work, I just start doing what I want to do, and I if come to something I don't understand or don't know how to implement, I just go and figure it out and then come back to coding.

Anyway, I didn't go with that because having an extra executable doesn't seem like the right idea here. So I thought about somehow having the type information at the runtime. if __TYPES__ is implemented, you can simply go through it, generate functions for each of the types in it, and write them to disk somewhere in the project and the developer can import those functions and use them and then they wouldn't need to use the bson library to construct the bson documents themselves. That seems to me to be more in line with what Gleam stands for because just generating a document ad hoc means that the developer can't guarantee the outcome at runtime, but this way Gleam's type system can actually help with that.

[lpil]

Whatever design we go for here we want it to be powerful enough to meet all our needs. The __TYPES__ proposal seems like it is not capable enough to generate all the boilerplate for this use case so I think it is not the right choice for us.

[massivefermion]

I've been thinking about this for the past few days, but it seems like I only have a vague idea and I cannot totally grasp it!
What if somehow types themselves could also be values and there was a reflection module in the standard library, something like dynamic but for types. So we could have:

import gleam/reflection

pub type User {
  User(username: String, score: Float)
}

reflection.kind(String) // "Primitive"
reflection.kind(User) // "Custom"
reflection.name(User) // "User"
reflection.fields(User) // [#("username", "String"), #("score", "Float")] or maybe [#("username", String), #("score", Float)] now that types are also values!

This would mean that Gleam should have a type for types(a kind maybe?). Or maybe two types, one for primitive types and one for custom types. Which would make things more clear and then you won't need the reflection.kind function.

Of course there is the complicated case of:

pub type User {
  Guest
  LoggedIn(username: String)
}

which I'm not sure how to handle right now.

Taking a look at this may help! It's very interesting but I'm not sure how we can adapt it to Gleam.

[lpil]

This is not possible as this metadata isn't available with Gleam. We do not store type information at runtime and if we were to add it there would be a large performance hit and FFI would be broken as we're not longer using the same data structures as the native language. It also goes against the Gleam philosophy of doing work at compile time rather than at runtime.

I think we need to look beyond asking what type a value has because more than that is needed to generate any boilerplate. We can't, for example, take the information TypeDefinition(name: "User", fields: [#("username", "String")]) and create a fn(Dynamic) -> Result(User, _) function with that.

[darky]

➕ 1️⃣ for having some reflection stuff in Gleam.
For example, https://github.com/gleam-lang/json package is little bit cumbersome in terms of writing code and requires of many gymnastics for simple JSON <-> Record transformation

[darky]

https://github.com/cdaringe/gserde LGTM for this case!