Syntax Extension: Use of this in Tokenize Handlers

[benrbray]

Thanks for creating this project! I'm banishing markdown-it from my project in favor of remark/micromark due to the emphasis on ASTs. It has been great so far!

I'm interested in learning how to build an extension to micromark, but the code is a bit difficult to navigate due to the absence of type annotations. I'm patching some of the types as I go, but I ran into a conceptual roadblock with the use of this inside of the handlers for syntax extensions.

Consider this code from micromark-extension-footnote. Similar code appears in several other built-in syntax extensions.

function tokenizeNoteEnd(effects, ok, nok) {
  // what type is `this`?
  var self = this

  return start

  function start(code) {
    var index = self.events.length
    var hasStart

    // Find an opening.
    while (index--) {
      if (self.events[index][1].type === 'inlineNoteStart') {
        hasStart = true
        break
      }
    }

    // istanbul ignore next - Hooks.
    if (code !== 93 || !hasStart) {
      return nok(code)
    }

    effects.enter('inlineNoteEnd')
    effects.enter('inlineNoteEndMarker')
    effects.consume(code)
    effects.exit('inlineNoteEndMarker')
    effects.exit('inlineNoteEnd')
    return ok
  }
}

I did a search of the micromark code to see where this function is bound, but no luck. How should I reason about what this function does? What type of object can I expect this to be bound to?

Edit: Based on usage, I suspect this is a Tokenizer. What can I assume about the contents of the current events stream (self.events) at the time this function is called?

Thanks for your help!

[wooorm]

Typically this refers to a Tokenizer.

But, in JavaScript this is “just” an extra (somewhat magic) parameter. In TypeScript it’s more tied to classes.
micromark is quite complex; this will mean different things in different places

[benrbray]

Thanks for the quick reply!

• Would it make sense to set the public-facing type for Tokenize to be the following (see e.g. this docs link)? (as of right now, actually, the type signature doesn't reflect usage)
  export type Tokenize = (this: Tokenizer, effects: Effects, ok: State, nok: State) => State;

• Also, in the same file, what is the purpose of setting nodeEnd.add = "after"? I see it used here, but this behavior is undocumented in the type signature for a Construct.

[wooorm]

mircomark is also relatively young. And it’s quite a new and ambitious project.

When I wrote it I focussed on tokenizing the core of markdown, but it was impossible to envision all potential extensions at the time. After it was released, I wrote a bunch of extensions. That gave the insights on how the APIs for extensions needed to be shaped.

Soonish I’ll go in and type + document how to make extensions, but for now it’s been a bit undocumented and private-ish, to prevent having to maintain quirky things for years to come.

For events: an event is a start or an end “event” of a token. It’s somewhat in between tokens and a syntax tree

[benrbray]

Got it, thanks! I'm already quite happy with what micromark has to offer compared to markdown-it, even it's a bit volatile for the time being. Keep up the good work!

[wooorm]

• For add: before|after, typically extensions will want to get precedence over existing markdown constructs. after can be used to invert that

• On Tokenize, I think you are correct.