Skip to content

Commit 34584f2

Browse files
gregsdennisgregsdennis
committed
json-e expressions
1 parent 2e88d72 commit 34584f2

File tree

2 files changed

+218
-0
lines changed

2 files changed

+218
-0
lines changed

‎_posts/2023/2023-11-27-json-e.md

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
---
2+
title: "JSON-e Expressions"
3+
date: 2023-11-28 09:00:00 +1200
4+
tags: [json-e]
5+
toc: true
6+
pin: false
7+
---
8+
9+
> _JSON-e is a data-structure parameterization system for embedding context in JSON objects._
10+
11+
At least that's how they describe it. My take would be that it's something of an amalgamation between [JSONLogic](https://jsonlogic.com/) and [Jsonnet](https://jsonnet.org/). It supports expressions, through which it can do a lot of the logic things that JSON Logic gives you, and it can performs data transformations using expressions, giving you a lot of what JSONata can do.
12+
13+
Their [docs](https://json-e.js.org/) are really great, and I recommend reading through those. It's not long, but it still does a good job of covering what JSON-e can do. I'll also be writing some docs for how you can use JSON-e in .Net.
14+
15+
This post is going to highlight some of the interesting aspects of the expression syntax that I discovered while implementing it. It's going to take a bit of setup, which is why this post is a bit longer than some of my others. So grab a drink and get comfy because it's gonna get fun.
16+
17+
## A brief introduction to JSON-e
18+
19+
To start, let's cover how JSON-e works at a high level.
20+
21+
The idea is pretty simple: you have a template and a context.
22+
23+
The context is just a JSON object which contains data that may be referenced from the template.
24+
25+
The template is a JSON value which contains instructions. Within those instructions can be expressions stored in JSON strings. These expressions are the focus for this post.
26+
27+
JSON-e takes the template and the context (JSON in) and gives you a new value (JSON out).
28+
29+
## What are expressions?
30+
31+
Before we get too deep into the weeds, some basic understanding of expressions is warranted.
32+
33+
JSON-e expressions are similar to what you might find in most programming languages, but specifically JS or Python. They take some values and perform some operations on those values in order to get a result.
34+
35+
The value space follows the basic JSON type system: objects, arrays, numbers, strings, booleans, and `null`.
36+
37+
You get the basic math operators (`+`, `-`, `*`, `/`, and `**` for exponentiation), comparators (`<=` and friends), and boolean operators (`&&` and `||`). You also get `in` for checking the contents of arrays and strings, `+` can concatenate strings, and you get JSON-Path-like value access (`.`-properties and `[]` indexers that can take integers, strings, and slices).
38+
39+
Operands which are not values are treated as context accessors. That is, symbols that access data contained in the context you provide. This allows expressions like `a.b + c[1]` where an expected context object might be something like
40+
41+
```json
42+
{
43+
"a": { "b": 1 },
44+
"c": [ 4, 5, 6 ]
45+
}
46+
```
47+
48+
## The context
49+
50+
While the context that you initially provide to JSON-e is a mere JSON object, as shown above, during processing the context is so much more.
51+
52+
There are some other keys that have default values, and they can be overridden by the object you provide.
53+
54+
For instance, the property `now` is assumed to be the ISO 8601 string of the date/time when evaluation begins. This property is used by the `$fromNow` operator. The effect is that this property is automatically added to the context so that if the template were to reference it directly, e.g. `{ "$eval": "now" }`, the result would just be the date/time string. However, if you were to include a `now` property in your context, it would override the implicit value.
55+
56+
```json
57+
{
58+
"a": { "b": 1 },
59+
"c": [ 4, 5, 6 ],
60+
"now": "2010-08-12T20:35:40+0000"
61+
}
62+
```
63+
64+
Furthermore, other operations, e.g. `$let`, provide their own context data that can override data in your context. But again, this is only within the scope of the operation. Once you leave that operation, its overrides no longer apply.
65+
66+
The net effect of all of this is that the context is actually a stack of JSON objects. Looking up a value starts at the top and works its way down until the value is found. In this way, you can think of that default `now` value as being a low-level context object with just the `now` key/value.
67+
68+
## Function support
69+
70+
Expressions also support functions, and you get some handy built-in ones, like `min()` and `uppercase()`. Each function declares what it expects for parameters and what its output is.
71+
72+
And just like operands for the expression operators, arguments to functions can be just about anything, even full expressions. This enables composing functions and passing context values into functions.
73+
74+
```json
75+
{ "$eval": "min(a + 1, b * 2)" }
76+
```
77+
78+
with the context
79+
80+
```json
81+
{ "a": 4, "b": 2 }
82+
```
83+
84+
will result in `4`.
85+
86+
## Functions as values
87+
88+
This is where it gets really cool.
89+
90+
I lied a little above when I said the value space is the JSON data types. _Functions are also valid values._ This enables being able to pass functions around as data. Many languages have this built in, but it's not part of JSON.
91+
92+
> Every implementation will likely be a bit different in how it makes this happen due to the constraints of how JSON is handled in that language, but JSON-e regards this as a very important feature.
93+
{: .prompt-info }
94+
95+
For example, I could have the template
96+
97+
```json
98+
{ "$eval": "x(1, 2, 3)" }
99+
```
100+
101+
In this case, `x` isn't defined, and it's expecting the user to supply the function that should run. The only requirement is that the function must take several numbers as parameters. A context for this template could be something like
102+
103+
```json
104+
{ "x": "min" }
105+
```
106+
107+
`min` is recognized as the function of the same name, and so that's what's called. You can also do this
108+
109+
```json
110+
{ "$eval": "[min,max][x](1, 2, 3)" }
111+
```
112+
113+
with the context
114+
115+
```json
116+
{ "x": 1 }
117+
```
118+
119+
This would run the `max` function from the array of functions that starts the expression, giving `3` as the result.
120+
121+
> Note that arrays and objects inside expressions aren't JSON/YAML values, even though it may look like they are. Because their values can be functions or reference the context, they need to be treated as their own thing: expression arrays and expression objects.
122+
{: .prompt-warning }
123+
124+
### In .Net
125+
126+
But, you may think, `json-everything` is built on top of the _System.Text.Json_ namespace, specifically focusing on `JsonNode`, and surely you can't just put a function in a `JsonObject`, right?
127+
128+
Wrong! You can wrap anything you want in a `JsonValue` using its static `.Create()` method, which means you can absolutely add a function to a `JsonObject`!
129+
130+
JSON-e functions are pretty simple: they take a number of JSON parameters and output a single JSON value. They also need to have access to the context.
131+
132+
That gives us a signature:
133+
134+
```c#
135+
JsonNode? Invoke(JsonNode?[] arguments, EvaluationContext context)
136+
```
137+
138+
In order to get this stored in a `JsonValue`, you could just store the delegate directly, but I found that it was more beneficial to create a base class from which each built-in function could derive. Also, in the base class I could define an implicit cast to `JsonValue`, which enables easily adding functions directly to nodes!
139+
140+
```c#
141+
var obj = new JsonObject
142+
{
143+
["foo"] = new MinFunction()
144+
}
145+
```
146+
147+
At certain points in the implementation, when I need to check to see if a value is a function, I do it just like I'm checking for a string or a number:
148+
149+
```c#
150+
if (node is JsonValue val && val.TryGetValue(out FunctionDefinition? func))
151+
{
152+
// ...
153+
}
154+
```
155+
156+
Embedding functions as data was such a neat idea!
157+
158+
> JSON-e has a requirement that a function MUST NOT be included as a value in the _final_ output. It can be passed around between operators during evaluation; it just can't come out into the final result.
159+
{: .prompt-info }
160+
161+
Also, kudos to the _System.Text.Json.Nodes_ design team for allowing `JsonValue` to wrap anything! I don't think I'd have been able to support this with my older Manatee.Json models.
162+
163+
## Custom functions
164+
165+
What's more, JSON-e allows _custom_ functions! That is, you can provide your own functions in the context and call those functions from within expressions! You want a modulus function? JSON-e doesn't provide that out of the box, but it does let you provide it.
166+
167+
In this library, it means providing an instance of `JsonFunction` (following the naming scheme of `JsonValue`, `JsonArray`, and `JsonObject`) along with a delegate that matches the signature from above.
168+
169+
```c#
170+
var context = new JsonObject
171+
{
172+
["mod"] = JsonFunction.Create((parameters, context) =>
173+
{
174+
var a = parameters[0]?.AsValue().GetNumber();
175+
var b = parameters[1]?.AsValue().GetNumber();
176+
177+
return a % b;
178+
})
179+
};
180+
181+
var template = new JsonObject
182+
{
183+
["$eval"] = "mod(10, 4)"
184+
};
185+
186+
var result = JsonE.Evaluate(template, context); // 2
187+
```
188+
189+
## Bringing it all together
190+
191+
And finally, the three aspects of JSON-e that I've discussed in this post come together in the most beautiful way.
192+
193+
- The context is a stack of JSON objects.
194+
- Functions are values.
195+
- Custom functions can be conveyed via the context.
196+
197+
_JsonPath.Net_ also supports custom functions in its expressions. To manage custom functions there, the static `FunctionRepository` class is used. At first, I wanted to use this same approach for JSON-e.
198+
199+
But once I figured out how to embed functions in data, I realized that I could just pre-load all of the functions into another layer of the context. Then the context lookup does all of the work for me! So now, when you begin the evaluation, the context actually looks like this:
200+
201+
```
202+
// top of stack
203+
- <user provided context>
204+
- { "now": "<evaluation start time>" }
205+
- { "min": <min func as a value>, "max": <max func as a value>, ... }
206+
```
207+
208+
Figuring this out was the key that unlocked everything else in my mind. How to include functions in a JSON object was the hard part. Once I realized that, the rest just kinda wrote itself.
209+
210+
## Introducing _JsonE.Net_
211+
212+
All of this is to say that I've had a fun time bringing JSON-e to .Net and the `json-everything` project.
213+
214+
I've learned a lot while building it, including aspects of functional programming, the whole putting-anything-into-`JsonValue` thing, and new ideas around expression parsing. I'll definitely be revisiting some of the other libs to see where I can apply my new understanding.
215+
216+
It's also been great working with the JSON-e folks, specifically [Dustin Mitchell](https://github.com/djmitche), who has been very accommodating and responsive. He's done well to create an environment where questions, feedback, and contributions are welcome.
217+
218+
This library is now available on Nuget!

‎_posts/2023/2023-11-09-updating-vocabs.md renamed to ‎_posts/2023/2030-11-09-updating-vocabs.md

File renamed without changes.

0 commit comments

Comments
 (0)