waltz integration

Thoughts on Waltz architectural diagrams

Use case

The initial use case is to go alongside 'boingy graph' as a new tab page. We can take the existing data model (i.e. the information contained about the nodes and edges behind the 'boingy graph') and use it to create a new diagrams which looks like the standard architectural diagram that you'd produce by hand, e.g. nodes are labelled applications and edges are labelled data flows between applications.

Caveats

digrams become hard to understand when there's more than 20-30 nodes (i.e. applications) in the graph. This needs to be played around with but maybe when the number of apps is greater than some ceiling, e.g. 30, we should not generate a diagram but instead display an error message 'Number of applications too large for architectural diagram'.

It would be nice if the new diagram could be contributed back to the Waltz open source project. There is a Bank process for doing that (but it's not particularly easy). Something to bear in mind.

Options

Both mermaid.js and d2 are options for the visualization engine. We should try both out for this use case and see what works best. Mermaid has the advantage being javascript and in browser. d2 has to my eye the easier to understand/ write syntax. Both are text formats for describing a diagram. Both are hierarchical in nature, i.e. you can contain parts of the diagram inside bounding boxes, which is often useful in architectural diagrams. Mermaid has weaker support for this than d2 where 'shapes' can contain other 'shapes'; i.e. it's a fundamental. Since we are starting with data - i.e. the data about nodes/ applications and edges/ flows, for both mermaid and d2, you'd have to write a 'compiler' e.g. a convertor that takes your data and outputs the mermaid/ d2 text format. This is where d2 comes in.

The purpose of d2

I wrote d2 in order to avoid having to write a new 'compiler' (above) for every new use case. d2 is essentially a datafied version of the d2 text format. Since it's data, it's possible to create other, higher level, data formats/ ways of expressing your graph on top of the basic dictim format. These are more convenient to work with for programmatic use cases. The key one here that we'll talk about is the dictim graph format which I think will be a good fit for the Waltz use case. I also designed dictim with AI in mind. i.e. you have some funadmentally nodes & edges data in some format (e.g. the data behind boingy graph), then tell AI about that data format and point it to dictim syntax documentation and it will write the code that does the conversion for you. This is covered on the dictim readme which I suggest you read at this point.

note:- As well as being a general compiler, dictim also supports converting d2 back into dictim format, but that's irrelevant for our use case.

The rest of this document is about dictim for the Waltz use case.

Installing

dictim is a command line tool, compiled for all main platforms. See installation instuctions. Inside the Bank, it's available through normal software request channels. Both dictim and d2 should be installed and available on the path. Check the version of dictim. The lastest is 0.9.1. This can be checked by running dictim -v or dict -v (the command line tool was renamed to 'dict' in version 0.9.1. 0.9.1 has a more consistent user interface for the command line tool and very thoroughly tested error handling; i.e. it's more production grade. If we find that we're on a different version, then Krish can sort us out on his return.

Example dictim

A very simple example with 3 nodes (i.e. apps) and 2 edges (i.e. data flows). dictim accepts both json and edn as input forms, but to keep it simple, we'll just talk about json from here on in.

diagram.json

[
    ["app1", "Application 1"],
    ["app2", "Application 2"],
    ["app3", "Application 3"],
    ["app1", "->", "app2", "fx trades, client ref data"]
    ["app1", "<->", "app3", "other trades"]
]

compile to d2

dict -c < diagram.json

and you'd get

app1: Application 1
app2: Application 2
app3: Application 3
app1 -> app2: fx trades, client ref data
app1 <-> app3: other trades

You'd then pass that into d2 to get back your svg, or in version 0.9.1 and as long as d2 is on your path, do it in one step with the -i option. dictim is well documented and has an extensive wiki so please read that. The -w option (watch) is good for when you're starting out with it and putting together some dictim in a text file in your editor of choice and want to see the resulting digram live. e.g. dict -iw.

dictim graph

As mentioned earlier I think dictim's higher level format may be the perfect fit for this use case. Here's an example of the dictim graph format

{
  "nodes": [
    {"id": "app1", "name": "Web Service", "dept": "Engineering", "cost": 15000},
    {"id": "app2", "name": "Database", "dept": "Infrastructure", "cost": 8000},
    {"id": "app3", "name": "Cache", "dept": "Infrastructure", "cost": 3000}
  ],
  "edges": [
    {"src": "app1", "dest": "app2", "data-type": "SQL queries", "frequency": "high"},
    {"src": "app1", "dest": "app3", "data-type": "cache requests", "frequency": "medium"}
  ],
  "node->key": "id",
  "node->container": "dept",
  "node-template": [
    [">", "cost", 10000], {"style.fill": "red"},
    ["=", "dept", "Engineering"], {"style.fill": "blue"}
  ],
  "edge-template": [
    ["=", "frequency", "high"], {"style.stroke-width": "3"}
  ]
}

documentation of this higher level format and it's usage is here.

As you can see from the example, the first advantage of this format is that it separates the data from how you want it visualized and styled in the diagram, so you can send in all your data i.e. all facts about nodes and all facts about edges rather than having to cut down to just the data required for labels that you want shown. This separation will allow for diagrams to be more dynamic. Imagine that we might ultimately have a series of dropdowns for the user to control aspects of the diagram, for example how the applications are grouped into clusters using the node->container key.

The second advantage is this format is that the various aspects of this diagram are separately controlled by different keys passed into a 'graphspec'. This means that we can add conditional styling to our diagram over time and dynamically rather than having to invest more and more time in a custom mermaid or d2 compiler built just for this waltz integration project.