Meta issue for discussion of this approach and API.

[leafpetersen]

This is an attempt to flesh out an idea that I've kicked around with a number of people to address the difficulty of interacting with untyped structured data in Dart 2. The canonical example is interacting with json, where the result of parsing is a large untyped blob of data that the user wants to be able to interact with in a typed way. This also arises with RPC calls in flutter, and also has come up with yaml.

See README for some examples of how this API would be used, as well as the test files.

Currently all that this provides is casting, but if the API turns out to work well, I would like us to explore using the schema to drive json and yaml parsing directly. That is, instead of parsing into untyped data and then casting, consider using the schema in the parser/RPC call/whatever to generate code with the expected reified type directly.

This meta issue is for discussion of the general idea of the API, as well as specifics around naming, missing functionality, etc.

I'll file a few specific sub-issues for additional discussions around the following:

• Should there be an incremental version of the API: Meta issue for discussion of an incremental API #2
• Should there be a lazy version of the API: Meta issue for discussion of a lazy API #3
• Should there be a debug version of the API: Meta issue for discussion of a debug API #4

[leafpetersen]

cc @munificent @rakudrama @vsmenon @efortuna @natebosch @matanlurey @nex3 for comment.

[efortuna]

cc @emshack fyi

[natebosch]

To make sure I understand correctly what does it accomplish to cast to int or String when using Keyed. Does it throw in the call to .cast(json) if the field is not of the right type?

const typed = cast.Keyed<String, dynamic>({
      "id": cast.int,
    });

[leafpetersen]

Keyed (other naming suggestions welcome) defines a "dependent" map schema. A normal MapCast has a single cast which it applies to all of the values in the input map range. A Keyed map applies different casts to different values in the range of the input map, controlled by the key. So

const typed = cast.Keyed<String, dynamic>({
      "id": cast.int,
    });

Is a cast which yields a Map<String, dynamic>, all of the keys of which are "id" and all of the values of which are integers. I think the point of your question is around the connection between the type arguments to Keyed and the casts in the range of the key map?

You could have written the above as:

  const typed = cast.Keyed<String, int>({
      "id": cast.int,
    });

In which case applying it would give you a map with same entries, but with reified type Map<String, int>.

However, incompatible types/entries are not allowed:

const typed = cast.Keyed<String, String>({ 
    "id": cast.int, // Static error, since Cast<int> is not a subtype of Cast<String>
  });

Basically, the second type parameter to Keyed should be a union type: the union of all of the types in the range of the argument map. Usually that will mean dynamic.

So for example, this schema describes maps which can have integer "id" fields and double "size" fields:

const typed = cast.Keyed<String, num>({ 
    "id": cast.int,
    "size": cast.double
  });

It will produce a Map<String, num>.

This is a schema which describes maps which can have the same fields as the above, or a String name field:

const typed = cast.Keyed<String, dynamic>({ 
    "id": cast.int,
    "size": cast.double,
    "name": cast.String,
  });

[natebosch]

Yeah I understood the way different keys can have different types, what I am wondering is exactly what value we get out of specifying fields that don't need any special handling.

Compare:

const typed = cast.Keyed<String, dynamic>({
  "id": cast.int,
  "name": cast.String,
  "age": cast.int,
  "size": cast.double,
  "extra": cast.List(cast.String),
});

var result = typed.cast(data);
var id = result["id"] as int;
var name = result["name"] as String;
var extra = result["extra"] as List<String>;

with

const typed = cast.Keyed<String, dynamic>({
  "extra": cast.List(cast.String),
});

var result = typed.cast(data);
var id = result["id"] as int;
var name = result["name"] as String;
var extra = result["extra"] as List<String>;

The first version has a lot of extra lines specifying the cast and there is no difference at the place where we read values. int and String always already had the correct type and would be fine with an as.

So, are we required to specify all fields we might see? And are we doing so in order for the exception to get thrown during the call to .cast(data) rather than the call to result["id"] as int;?

[leafpetersen]

Fair point. I think there's value in being able to lock down the schema so that you catch things with unexpected fields, but there's no reason not to allow a "default" cast which gets applied to any fields not found in the map. Then the question is, which should be the default behavior - allow extra fields (with default cast of cast.any) or lock down the fields?

[leafpetersen]

Filed #5 and #6

[vsmenon]

Nice!

Perhaps simple add:

String jsonData = ...;
var result = typed.parse(jsonData);

if that allows us to save memory allocations when there is native JSON support. E.g., invoke V8's JSON.parse and destructively change the resulting object to a typed Dart one.

[itsjoeconway]

Does this approach eventually end up at JSON Schema? Is the purpose to fail at parse-time instead of failing when accessed later? (Is that even feasible for, say, a large list?) Is the additional value over List.cast<E> and Map.cast<K,V> that you can more deeply describe the schema?

It appears one has to define types and names in at least 3 different places. In two of those places, the names are stringly-typed. This was an approach in Objective-C (except a type defined its own schema as a static 'property'). It was OK, but not ideal.

I can't help but think this is a bandaid on an underlying language problem. Types are getting tighter, but the language tools to specify types aren't catching up.

[leafpetersen]

Does this approach eventually end up at JSON Schema?

I don't think so, but I could be wrong. The initial goal is to provide a way to do deep casts easily. That is, in Dart 1, you could take a large blob of full untyped data and simply treat it as typed, and it would just work.

List<Map<int, List<String>>> l = blob_of_untyped_data;

You can't do that in Dart 2, and we see this causing a lot of user confusion. My goal was to try to provide similar convenience. So now you just do:

const type = cast.List(cast.Map(cast.int, cast.List(cast.String))); 
var l = type.cast(blob_of_untyped_data);

That is, you basically change the < to (, and add a .cast at the end, and you're done.

Is the purpose to fail at parse-time instead of failing when accessed later?

No. The current implementation is eager, but one question is whether it's important to have a lazy API. It should be easy enough to do in any of a number of different ways.

Is the additional value over List.cast and Map.cast<K,V> that you can more deeply describe the schema?

Yes. I see a lot of people writing (struggling to write) repeated code that looks like:

  var typedData = (untyped_blob as List).map((a) => (a as Map).cast<String, int>())

The problem is that because casts are shallow, you either need to do the casts as you unspool the data, or you need to use a variety of techniques to roll your own eager deep casts.

This library provides a single interface to do that.

It appears one has to define types and names in at least 3 different places. In two of those places, the names are stringly-typed.

I don't follow this, what do you mean? In the example above, I wrote the "type" exactly once, and didn't use any strings?

[itsjoeconway]

I'm going to trial this in the open_api package to better understand and have better feedback. OpenAPI a very large and diverse JSON document, and it has been challenging to fit it into Dart 2.

First thing I've ran into is that I have to specify every key. There are top-level values that are Map<String, dynamic>. I am unable to find (and it sounds silly to say) a cast.dynamic or similar.

final schema = cast.Keyed<String, dynamic>({
      "swagger": cast.String,
      "host": cast.String,
      "basePath": cast.String,
      "schemes": cast.List(cast.String),
      "consumes": cast.List(cast.String),
      "produces": cast.List(cast.String),
      "security": cast.List(cast.Map(cast.String, cast.List(cast.String))),
      "info": /* what goes here? It is a Map<String, dynamic> */
    });

edit: There is clearly cast.Keyed<String, dynamic>, but does this mean I have to describe the entire document in one schema?

[itsjoeconway]

I have to bounce around a few meetings here shortly, but I've included a peek so far. What I meant by specify the type/name multiple times should be apparent from below (unless I'm totally doing this wrong). For example, I specify the field List<String> schemes in my type, the key-value pair "schemes": cast.List(cast.String), and the assignment schemes = typed["schemes"]. I've left some comments in line.

// 'APIObject' base class requires 'decode' implementation, allows deeply nested
// 'APIObject's to be parsed via their 'decode' method.
class APIDocument extends APIObject {
   String version;
   List<String> schemes;

   ... snip ...
   // JSONObject is a MapMixin that has decoding and encoding methods for various types.
   // A JSON string is decoded such that each Map is revived as a JSONObject.
   void decode(JSONObject object) {
    super.decode(object);

    final schema = cast.Keyed<String, dynamic>({
      "swagger": cast.String,
      "host": cast.String,
      "basePath": cast.String,
      "schemes": cast.List(cast.String),
      "consumes": cast.List(cast.String),
      "produces": cast.List(cast.String),
      "security": cast.List(cast.Map(cast.String, cast.List(cast.String))),

       // Nested objects have their own schema that they define
       // Is this where I make the argument for inheriting static methods? :)
      "info": APIInfo.keyedSchema,
      "tags": cast.List(APITag.keyedSchema),
      "paths": cast.Map(cast.String, APIPath.keyedSchema),
      "responses": cast.Map(cast.String, APIResponse.keyedSchema),
      "parameters": cast.Map(cast.String, APIParameter.keyedSchema),
      "definitions": cast.Map(cast.String, APISchemaObject.keyedSchema),
      "securityDefinitions": cast.Map(cast.String, APISecurityScheme.keyedSchema)
    });

    final typed = schema.cast(object);

    // This looks fairly clean.
    version = typed["swagger"];
    host = typed["host"];
    basePath = typed["basePath"];
    schemes = typed["schemes"];
    consumes = typed["consumes"];
    produces = typed["produces"];
    security = typed["security"];

   // This is what I need to figure out next. There was previous behavior to handling 
   // decoding deep collections into their associated Dart types that now has to get 
   // evaluated. 
    info = object.decodeObject("info", () => new APIInfo());
    tags = object.decodeObjects("tags", () => new APITag());
    paths = object.decodeObjectMap("paths", () => new APIPath());
    responses = object.decodeObjectMap("responses", () => new APIResponse());
    parameters = object.decodeObjectMap("parameters", () => new APIParameter());
    definitions =
        object.decodeObjectMap("definitions", () => new APISchemaObject());
    securityDefinitions = object.decodeObjectMap(
        "securityDefinitions", () => new APISecurityScheme());
  }
  ... snip ...
}

[leafpetersen]

'm going to trial this in the open_api package to better understand and have better feedback. OpenAPI a very large and diverse JSON document, and it has been challenging to fit it into Dart 2.

@joeconwaystk Thanks for looking at this - I was hoping to get this exact kind of feedback. I have no experience with real json applications. Even if this approach doesn't work out for that use case, the process of seeing where it falls down will be helpful for finding the right solution.

I am unable to find (and it sounds silly to say) a cast.dynamic or similar.

Just added this as cast.any, and added the ability to specify a default for Keyed. You should be able to do this now:

final schema = cast.Keyed<String, dynamic>({
      "swagger": cast.String,
      "host": cast.String,
      "basePath": cast.String,
      "schemes": cast.List(cast.String),
      "consumes": cast.List(cast.String),
      "produces": cast.List(cast.String),
      "security": cast.List(cast.Map(cast.String, cast.List(cast.String))),
      "info": cast.Map(cast.String, cast.any) // Could also probably be just cast.any if you don't want to validate
    });

[leafpetersen]

@joeconwaystk

I don't know what all of the types involved in your example above are, but below is a quick take on how you might structure it. A couple of comments:

• I'm not sure this is the best way to structure this, but I think it is a way
• Agreed that the redundancy between class types and schema types is annoying
  • To avoid this you would either need language/compiler magic (reflection), or codegen
• I think this suggests value in allowing Keyed schemas to be merged to support the "inheritance" behavior more cleanly.
• I've replaced your JSONObject type with Map<String, dynamic>, not sure if it was important to your example?

class APIObject {
  final String apiObjectField;
  static const Map<String, cast.Cast<dynamic>> fieldDescriptors = {
    "apiObjectField": cast.String,
  };
  static cast.Cast<APIObject> schema =
      cast.Apply((a) => APIObject.fromJson(a), cast.Keyed(fieldDescriptors));

  APIObject.fromJson(Map<String, dynamic> fields)
      : apiObjectField = fields["apiObjectField"];

  static APIObject decode(Map<String, dynamic> object) => schema.cast(object);
}

class APITag {
  static cast.Cast<APITag> schema =
      cast.Apply((a) => APITag.fromJson(a), cast.int);
  APITag.fromJson(int x);
}

class APIInfo {
  static cast.Cast<APIInfo> schema =
      cast.Apply((a) => APIInfo.fromJson(a), cast.int);
  APIInfo.fromJson(int x);
}

class APIDocument extends APIObject {
  String version;
  String host;
  String basePath;
  List<String> schemes;
  List<String> consumes;
  List<String> produces;
  List<Map<String, List<String>>> security;
  APIInfo info;
  List<APITag> tags;

  static Map<String, cast.Cast<dynamic>> fieldDescriptors = {
    "swagger": cast.String,
    "host": cast.String,
    "basePath": cast.String,
    "schemes": cast.List(cast.String),
    "consumes": cast.List(cast.String),
    "produces": cast.List(cast.String),
    "security": cast.List(cast.Map(cast.String, cast.List(cast.String))),
    "info": APIInfo.schema,
    "tags": cast.List(APITag.schema),
  }..addAll(APIObject.fieldDescriptors);

  static cast.Cast<APIDocument> schema =
      cast.Apply((a) => APIDocument.fromJson(a), cast.Keyed(fieldDescriptors));

  APIDocument.fromJson(Map<String, dynamic> fields)
      : version = fields["swagger"],
        host = fields["host"],
        basePath = fields["basePath"],
        schemes = fields["schemes"],
        consumes = fields["consumes"],
        produces = fields["produces"],
        security = fields["security"],
        info = fields["info"],
        tags = fields["tags"],
        super.fromJson(fields);

  static APIDocument decode(Map<String, dynamic> object) => schema.cast(object);
}

[nex3]

This seems super useful! It looks like a great tool for doing the sort of inline List<Map<int, List<String>>> casts you're talking about, and I bet it could be expanded beyond that as well. That said, I'm worried that it's not going to be powerful enough to work for the more advanced cases I've run into. The biggest difficulties I see on the horizon are:

• Handling source spans. This doesn't matter when dealing with JSON, but it's super important for YAML to produce SourceSpanFormatExceptions when casts fail so that users know where in their file they made a mistake. It's also important that, even when casts succeed, source spans for the parsed values are available so that they can be stored and used if errors occur later on.

  The way the yaml package handles this is to return YamlMaps and YamlLists that expose SourceSpan span fields and fields such as List<YamlNode> nodes. These nodes fields contain the same values as the original collection, except that non-collections are wrapped in the YamlScalar class which has a SourceSpan span field.

  As written, the cast library would ignore these spans and just return their underlying values, which means users would have no access to their source spans.

• The Keyed cast relies heavily on implicit casts from dynamic to work gracefully. The trajectory of Dart has generally been towards adding warnings for implicit behavior like this, which will make this pattern a lot more painful. Even if we decide that's not an issue, this behavior creates a class of potential errors which only appear at runtime—for example, in your code sample above, if I changed List<String> schemes to List<int> schemes, it would pass static analysis and fail at runtime, not because the untyped data was wrong but because the code's types were incorrect.

• I like that this is extensible, but I'm worried that the barrier to entry for creating custom casts is too high. In the configuration-parsing code I've written, it's very common to want to express things like "cast to a string and parse it as a Version", and the code necessary to do that is pretty terse. Having to go through the boilerplate of creating a new class for every piece of behavior I want to factor out would be a heavy cost.

  It's worth looking at matcher as an example. It requires new matchers to be defined as classes, and as a consequence most code just uses the predicate() matcher instead of defining custom matchers, even though predicate() provides much worse output when it fails.