User extensions

[tavin]

I have a couple of ideas/wishes related to markdown parsing and rendering, and it seems a MyST syntax extension might be the natural (only?) way to realize them. However the barrier to entry is perhaps rather high. Do I submit a MEP? Do I need to make contacts in multiple organizations for that?

I can think of a couple ways to make it much easier for a student/hobbyist to get started on something like this:

• Processing of enable_extensions could treat a dotted string as a reference to an external object which satisfies an extension interface.
• There could be a separate user_extensions list with like behavior.

From what I've seen I suppose the interface would just be a callable that receives the markdownit parser.

[chrisjsewell]

See my answer in executablebooks/MyST-Parser#632 (comment)

[agoose77]

I'm happy to discuss user extensions more broadly, but then it would be best to open a separate discussion.
This was a specific question, that already had a specific answer

Personally I felt that the issue was general enough that it warranted further context. I agree that any additional discussion might be better suited in a dedicated thread, but this issue will still appear in the search given the content, so that's where I pitched my reply. It actually took me a second pass to realise that there was a specific reference to myst-parser configuration that scoped it to that tool!

and also I would suggest this is not quite correct; I created markdown-it-py with the exact purpose of having compatibility between python and js, and indeed all of the markdown-it-xxx package in this organisation derive from extensions in mdit-py-plugins

I think this is semantics. A third-party user can't currently write a single blob of code that extends the MyST syntax across all frontends, and that's what I'm referring to here. The fact that we have a Python port of markdown-it is fantastic, because it means adding new syntax is a lot easier.

[chrisjsewell]

A third-party user can't currently write a single blob of code that extends the MyST syntax across all frontends, and that's what I'm referring to here.

by front ends, do you mean HTML/LaTeX/JATS, etc?
The way to do that is to have a defined set of node types, which is what pandoc and sphinx/docutils have, although sphinx also allows you to introduce new nodes, which you then have to specify builder specific rendering logic (or have fallbacks)

In terms of source to AST parsing; if the spec is clear enough anyone can implement it
As I've mentioned before, djot has at least five different implementations, over four different programming languages. They've all managed to implement the spec, and change with it as its evolved

[agoose77]

by front ends, do you mean HTML/LaTeX/JATS, etc?

Ugh, you go to one Jupyter workshop and "frontend" becomes every third word 😆

The way to do that is to have a defined set of node types, which is what pandoc and sphinx/docutils have, although sphinx also allows you to introduce new nodes, which you then have to specify builder specific rendering logic (or have fallbacks)

Yes, and that's what I was touching upon with the idea that arbitrary roles and directives are soft extension targets that we can more easily support across renderers, as they have graceful (enough) fallbacks and any extension mechanism can be designed such that it maps roles/directives to a custom renderer. Whereas, custom parsing is a whole other kettle of fish.

In terms of source to AST parsing; if the spec is clear enough anyone can implement it

That's my hope for the spec — parsers can come and go as long as they produce our AST and don't hurt anyone! :P

[tavin]

Well even being able to add just roles and directives in a pluggable way would be helpful. Indeed it seems clear enough that rendering can fall back to code blocks or divs with a class.

As the scope of this discussion has grown, let me add a couple remarks.

Personally I don't care if I have to reimplement my custom code across different parse/render toolchains. That's normal.

I also don't care if I have written an extension based on a specific platform (e.g. the markdownit parser) and then upstream changes to a new platform. In my view, as a custom extension author it's my task to adapt my code, and there is no interface contract.

So whether we're talking about roles/directives or my original ask, I'd rather have something limited by these 2 caveats than not have it at all.

[tavin]

See also jupyter-book/jupyterlab-myst#115 which opens a pathway to custom directives. It would become possible to write a separate labextension adding directive specs when the myst parser is called.