The mdx rewrite

[ckohen]

Context

@shaydewael mentioned in #5374 that mdx is now supported by the render engine. Though the change was removed from that PR to make the diff cleaner, the switch seems to be coming hard and fast.

There's a small (but very active in this repo) group of people that did this conversion in the past during a hackweek, you can see the repo here. It allowed us to make a lot of usability changes and generally was better for QOL. Seeing some of those changes come to the live site would be the dream.

Questions / Concerns

1. Does support for mdx mean that we can contribute our own components in this repo? I.e. the first thing I'd do is copy the Snowflake conversion svg component to replace the picture. Makes it scale better with the page and the text selectable. Some other things I'd like to see early on is replacing the hijacked blockquote info blocks with the Alert component and the improved RouteHeader component with the badges for supporting audit log header and being an unauthenticated request.
2. Given the above, are we allowed to add deps should they become necessary? It seems the renderer loads this repo as a dependency which would install any required deps.
3. Components we contribute to the repo would have to be jsx unless we commit the tsx and built jsx files since the repo is pulled in from github and not npm. This seems....not fun to say the least.
4. The only thing we'd be missing is the ability to build the site locally to actually test these changes, which unfortunately makes the dev cycle that allows our own components practically non-existent.

Timeline

1. Convert the entire repo to mdx in a single PR that doesn't touch anything else. This should be done just after a merge wave so the number of PRs with conflicts is reduced. (converting everything isn't strictly necessary, but it doesn't hurt and makes diffs when conversions would've happened much better)
2. This conversion requires an update to the link checking CI workflow, this should occur in the above PR.
3. Convery the table that shay originally wanted to convert to verify mdx is working
4. Figure out whether we can somehow use tsx for custom components.
5. Convert the snowflake image to a component to verify components work.
6. Figure out a way to build the site locally so we can have an actual dev cycle (and PR previews) or move the docs portion of the site entirely to this repo, keeping the part of the dev portal that requires backend access (applications etc...) private and just linking to them.

[shaydewael]

These are my own thoughts, but they aren't set in stone or the thoughts of other apps team members....

• At first, I don't think we'd support component contributions. For a migration like this, it'd be easier to start off small—things like converting alert components like you mentioned (which I can do). I really really do not want the actual conversion to mdx become a huge project, or else it'll likely be pushed off for a while.
• The component code also lives on our side, and there's not a good way to opensource the entire docs site at the moment. I just don't think that's in the cards right now unfortunately, even if it'd be nice to see. There isn't clear enough separation between the documentation code and the rest of the dev portal code on the backend for us to opensource it in an easy way, and keeping it together at the moment has the benefit of the being able to pull in components from Discord (buttons, select menus, etc etc) with a relatively small amount of work.
• If/when we did start supporting more components, it'd be on me to document those in a reference somewhere within this repo so there's understanding of existing practices, and alignment on how a "component library" for the docs site might scale (if that's the intention).
• Short-term, I think we want to avoid any additional dependency contributions unless there's a really compelling reason. There's a possibility we refactor parts of the documentation, so we'll want to wait to make non-incremental changes until there's more clarity around that. That would also be a time where we could think about potential strategies to opensource the documentation.
• I do think we can pull some of the changes the hackweek site showcases (and I'm looking into it). If there are any specific aspects besides the ones you've highlighted already, let me know. Things that are going to be far more complicated are around rendering navigation or anything else around the frame/container of the site. One thing I'm trying to pull in short-term are restyling the headers for method names to make the get/post/patch/put/delete elements more visually apparent. I'm also quite keen on the other sorts of badges around authentication/audit log/etc but I think that won't immediately come.
• Also...I'm working on some major updates to the Introduction page which is going to introduce some basic components to make it easier to display certain kinds of information that our docs are not good at right now. This is already in-progress, and I can share more info on it soon. That is to say:
  1. components do work—I've been testing them on my own environment and have already built out some prototypes. I also build the docs locally when writing new content, so I'm sure the table in that was originally in the Gateway PR does work.
  2. the transition to mdx is pretty imminent unless there's strong pushback (so yes, the hard and fast is probably accurate)

[ckohen]

This is all kind of what I was expecting. My one concern is components for mdx are a bit different than the highjacked md rendering engine components we're used to. Generally speaking you import the components at the top of a file like any other language and place them where you need them. So yes we'd need documentation to point at the file paths that will exist. However, I don't see how that stops us from making a components directory in this repo and importing from it (the deps thing was looking a little far out). Either way, that directory should probably exist, either to house some components and a readme for how to use components, or just the readme for how to use the components that exist only at build time.

[shaydewael]

So it's unideal but from what I understand: currently for MDX we still do a mash up of the existing parser and "real" MDX—the parser supports MDX components/their general syntax on our end, but it's still parsed in a semi-hijacky way rather than components being imported like traditional MDX (at least from what I've seen). How I've been thinking about it is kind of like...global MDX components rather than per-file imports, with some custom parsing logic required. In the future, we should of course standardize on MDX or something similar as it's going to make it so much easier for everyone involved (I personally love Stripe's Markdoc but that's a me opinion which i don't think is widely shared so it almost certainly won't happen lol). Take how it works with a grain of salt though—I need to get past the early prototyping phase to make sure I understand how it works and what it is and isn't capable of in its current state, which I'll eventually follow up on in some form.

And yeah I can do the readme at first for the components (and an example/image of render) because those need to be documented anyway, then can re-evaluate as we move forward and evolve the documentation.

Sorry that isn't an ideal answer and perhaps dampers some of the benefits of moving to MDX. I do hope and (truly) think we'll move to a better/more consistent standard at some point in the not-so-distant future, but that just isn't now :/ I think aligning with general MDX patterns makes sense where we can, so when we eventually fully transition, it'll be a pretty low-lift and won't radically alter the docs.

[ckohen]

The global component analogy is pretty helpful (that'll be great in the readme). Just wondering if you've tried importing a local component? Maybe it works in both ways 🤞

[shaydewael]

I haven't tried it yet, but I can get back to you on it sometime pretty soon. I was poking around in all of this quite a while ago, so I don't have the same context that I did a month ago 😓 buttt i'll have a pretty big focus on some of this in the next couple of weeks so I can give more insight then.

[MindlessSteel]

I'm not a contributor but I like what you've written so far and yes I believe it should be kept to a small group of people and get it proven and validated before any further adoption can take place