diff --git a/.gitignore b/.gitignore index 54ef78ed..9776b088 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,6 @@ _build # Python files __pycache__ + +# Jupyter Server +.ipynb_checkpoints/ diff --git a/xx-text-based-diagrams-in-markdown/example.dot.svg b/xx-text-based-diagrams-in-markdown/example.dot.svg new file mode 100644 index 00000000..f46e1a15 --- /dev/null +++ b/xx-text-based-diagrams-in-markdown/example.dot.svg @@ -0,0 +1,38 @@ + + + + + + +LR + + + +chicken + +chicken + + + +egg + +egg + + + +chicken->egg + + + + + +egg->chicken + + + + + + diff --git a/xx-text-based-diagrams-in-markdown/example.mermaid.svg b/xx-text-based-diagrams-in-markdown/example.mermaid.svg new file mode 100644 index 00000000..366dd670 --- /dev/null +++ b/xx-text-based-diagrams-in-markdown/example.mermaid.svg @@ -0,0 +1,2 @@ + +
chicken
egg
 diff --git a/xx-text-based-diagrams-in-markdown/example.plantuml.svg b/xx-text-based-diagrams-in-markdown/example.plantuml.svg new file mode 100644 index 00000000..2a88ddd1 --- /dev/null +++ b/xx-text-based-diagrams-in-markdown/example.plantuml.svg @@ -0,0 +1 @@ +ChickenEggmany1 diff --git a/xx-text-based-diagrams-in-markdown/text-based-diagrams.md b/xx-text-based-diagrams-in-markdown/text-based-diagrams.md new file mode 100644 index 00000000..f5c831e3 --- /dev/null +++ b/xx-text-based-diagrams-in-markdown/text-based-diagrams.md @@ -0,0 +1,557 @@ +--- +title: Adopting a text-based diagram syntax in Jupyter Markdown +authors: (alphabetically) Nick Bollweg (@bollwyvl) +issue-number: 100 +pr-number: 101 +date-started: 2021-03-14 +--- + +# Summary + +This proposal encouraged _Project Jupyter_ to adopt a _text-based diagram_ format in +_Jupyter Markdown_. Easy-to-write diagrams should render consistently, accessibly, and +securely across flagship _Jupyter tools_ like _JupyterLab 4_, _Notebook 7_, _nbconvert_ +and _nbviewer_, in addition to the platforms _Project Jupyter_ uses to collaborate, like +_GitHub_, _HackMD_, and _Discourse_. This proposal recommends adoption of +[_MermaidJS_][mermaidjs], with the _fenced code block_ syntax inherited from +_GitHub-flavored Markdown_. + +[mermaidjs]: https://github.com/mermaid-js/mermaid + +# Motivation + +Embedded _text-based diagrams_ within _Jupyter Markdown_ will provide a way to augment +plain- and rich-text narrative when working **in** _Jupyter tools_, as well as **on** +_Jupyter tools_, such as _GitHub_, _Discourse_, _HackMD_, and other common collaboration +platforms. + +Beyond Jupyter tools, selecting a widely-adopted implementation and syntax will require +less re-learning of syntax and skills. + +# Guide-level explanation + +With the addition of [_MermaidJS_][mermaidjs] to _JupyterLab 4_, _Jupyter Notebook 7_, +_Jupyter NBConvert_, and _Jupyter NBViewer_, _Jupyter_ joins +[GitLab](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3711), +[HackMD](https://github.com/hedgedoc/react-client/issues/257), +[GitHub](https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/) and +[many other platforms](https://mermaid.js.org/ecosystem/integrations.html) in natively +supporting a portable, interoperable _GitHub-Flavored Markdown_ (GFM) syntax for +_text-based diagrams_ to visually describe complex topics. + +## What can I draw? + +Today, you can start creating the following diagrams in _Jupyter Notebooks_ or _Jupyter +Markdown_ files: + +- _Class Diagrams_ that show the relationships between software components +- _Entity Relationship Diagram_ taht show the relationships between data models +- _Flowchart_ that show decision processes +- _Gantt Charts_ that show the relationship between tasks over time +- _Pie Charts_ that show the relative size of different values +- _Requirement Diagrams_ that show the chain of relationships from _what a user needs_ + to _what we should build_ +- _Sequence Diagrams_ that show the exchange of information between different people, + systems, or processes +- _State Diagram_ that show the possible states a system can be in, and how they might + change +- _Timeline Diagrams_ that show events over time +- _User Journey Diagrams_ that show the cycle of events over time + +...and they should _mostly_ render the same way everywhere. + +_MermaidJS 10_ also includes a number of new diagrams, not available on earlier major +versions. You can author these today, but they won't work on other platforms _for a +while_: + +- _Gitgraph (Git) Diagram_ that show the sometimes-bewildering history of complex + _distributed version control_ operations +- _Mindmaps_ that show the relationship between different ideas +- _Timeline Diagrams_ that show events over time + +Finally, a number of **experimental** diagrams and layout engines are available, but may +change significantly in the future: + +- _C4C Diagram (Context) Diagrams_ that show different facets of software architectures +- _ElkJS_ that extends the _Flowchart_ syntax to expand + +## How do I draw it? + +The [MermaidJS Documentation](https://mermaid.js.org/intro/) provides full examples of +all the supported diagram formats. But, mostly, try it out inside your _Jupyter tools_! + +## Why _text-based diagrams_? + +Unlike diagrams authored in heavyweight drawing programs, or as the output of code, +_text-based diagrams_ are easy to create by hand, and are usually readable when they +_can't_ be rendered fully. This helps _everyone_ understand visual concepts, including +those that make use of _assistive devices_ like _screen readers_. By doing this +rendering right in your browser, your ideas become visual as quickly as possible: +combined with the _Jupyter Collaboration_ features introduced in _JupyterLab 3.1_, and +refined in _JupyterLab 4_, this reactivity can be shared with your whole time. + +By choosing this widely-implemented, portable syntax, we hope the all-important _muscle +memory_ mastery developed while working in _Jupyter tools_ are applicable to other +digital platforms, and hopefully will improve the communication of _Project Jupyter_ +itself. + +## Why _MermaidJS_? + +While many other [_text-based diagram_](https://text-to-diagram.com) formats exist, +_MermaidJS_ enjoys: + +- broad adoption across many tools and platforms with a consistent syntax +- a relatively simple and readable format +- a wide number of software-oriented and general-purpose diagrams +- runs entirely within the browser + +# Reference-level explanation + +## _MermaidJS_ Details + +_MermaidJS_ is _free software_, distributed under the permissive +[MIT License](https://github.com/mermaid-js/mermaid/blob/develop/LICENSE) and adheres to +a +[code of conduct](https://github.com/mermaid-js/mermaid/blob/develop/CODE_OF_CONDUCT.md). + +The codebase has been actively publishing releases since 2014, with over 400 historical +contributors, with a relatively smaller core of active contributors. + +The `mermaid` package itself is distributed on +[`npmjs.com`](https://www.npmjs.com/package/mermaid) as well as _content delivery +networks_ (CDN). + +At install/run time, aside from its own assets, it entails a net addition of `dagre-d3` +and the full `d3` metapackage. Some advanced features use the `cytoscape` and `elkjs` +_rendering engines_, but are only loaded when needed. + +## Unparseable Diagrams + +As a _fenced code block_, without broad syntax highlighting support, Jupyter clients +that do not support _MermaidJS_ at all would fall back to the default typography and +color chosen for code content by the client and/or user. As _text-based diagrams_ are +usually written by hand, an unparseable diagram was _at least_ readable to the author at +time of writing, and likely their audience will be able to glean _some_ meaning from the +text itself. + +It is therefore recommended that unparseable diagrams be presented as plain code text, +preserving the whitespace of the source document, with an option to see any parser +errors. Ideally, the text would be syntax highlighted, but as noted elsewhere, this is +somewhat non-trivial. + +## Misparsed Diagrams + +Today, only Jupyter clients that have been extended (by e.g. `jupyterlab-markup`) will +render _MermaidJS_ diagrams with the _fenced code block_ syntax, using whatever version +is currently included in the extension (e.g. `8.14`). Diagrams written for those version +are _very likely_ to continue being parsed by _MermaidJS 10_, but there are certainly +cases where the reverse is not true, as in the case of some of the experimental diagrams +not listed above. + +## Alternative Implementations + +If a Jupyter client is unable to adopt MermaidJS, but can offer a best-effort, +grammar-compatible implementation that will draw _something_, this is preferable to, but +should not exclude, presenting the diagram as plain text. + +## _JupyterLab_ PR #1402 + +[jupyterlab/jupyterlab#1402](https://github.com/jupyterlab/jupyterlab/pull/14102) +proposes adding `mermaidjs 10` to the default _Markdown_ rendering engine, `marked`. As +_JupyterLab 4_ is the upstream of _Jupyter Notebook 7_, landing this PR would mean only +a few configuration files would have to change. + +### Implementation + +Implementing `mermaid` with _fenced code blocks_ means _every_ code block is passed +through an extra check for the `mermaid` syntax marker. When found for the first time, +the core `mermaid` bundle (and its transient dependencies) are lazily loaded, and each +extra kind of diagram loads an additional grammar. + +This increases the complexity of the implementation a fair amount, and uses +_asynchronous rendering_ more aggressively than was previously used. + +### Size + +While lazily-loaded, and having no appreciable impact on the _time-to-first +interaction_, these _MermaidJS_ assets are **not small** when installed, or when served +to the browser. The following table compares some relative sizes of the full +distribution of all possible files that _could_ be loaded by a _JupyterLab_ user: + +| | uncompressed (mb) | gzip (mb) | +| :----------- | ----------------: | --------: | +| `4.0.0a46` | 7.02 | 1.98 | +| `08f4d6e` | 9.76 | 2.78 | +| only `elkjs` | 1.43 | 0.41 | + +> - the _uncompressed_ column is what a user _typically_ sees when loading the +> application. +> - the _gzip_ column _roughly_ can be seen as the impact on the as-distributed wheel on +> PyPI + +`elkjs` is called out explictly, due to its size. Likely the state-of-the-art in terms +of _auto layout_ tools, it has potential _well_ beyond many other engines for complex +diagrams, but is currently hidden behind an experimental feature flag, almost certainly +not enabled on other platforms. It may be sensible to omit it from the initial +distribution. + +### Security + +To avoid _cross-site-scripting attacks_, the `marked` API is used with all available +security features, and further applies sanitiziation. Unfortunately, this forces the +delivery of diagrams as base64-encoded +[Data URIs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs) +as the `src` attribute of `` tags. This means that all interactive _MermaidJS_ +features, like hypyerlinks, tooltips, and cross-diagram configurable styling, are +unavailable. + +An alternate approach could render these inside restrictive `