Marko 5 ESM Loading/Registering

[adamchal]

Marko Version

5.15.1

Details

Trying to render a Marko template to HTML using Node in a module (ESM) context. Minimal repo showcasing the issue.

Using marko.load() to load the Marko template

import marko from "marko";
(async () => {
  const template = marko.load("./test.marko");
  const rendered = await template.render();
  console.log(rendered.getOutput());
})();

Expected Behavior

HTML output of test.marko logged to the console.

Actual Behavior

WARNING!!
Using `marko/compiler` has been deprecated, please upgrade to the `@marko/compiler` module.
  at node:internal/modules/cjs/loader:1095:14

/marko-5-esm-issue/components/foo.marko:1
div -- foo
       ^^^

SyntaxError: Unexpected identifier
    at Object.compileFunction (node:vm:352:18)
    at wrapSafe (node:internal/modules/cjs/loader:1025:15)
    at Module._compile (node:internal/modules/cjs/loader:1059:27)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1124:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:816:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:93:18)
    at Object.<anonymous> (/marko-5-esm-issue/test.marko:9:36)
    at Module._compile (node:internal/modules/cjs/loader:1095:14)

Possible Fix

However, if you marko.load the component before loading the template, it works.

import marko from "marko";
(async () => {
  marko.load("./components/foo.marko"); // now it works
  const template = marko.load("./test.marko");
  const rendered = await template.render();
  console.log(rendered.getOutput());
})();

But, since components can reference other components, you’d have to load them in order of least dependent to avoid the error.

I have not found clear documentation on how to do this in Marko 5. Previously, in Marko 4, I would require('marko/node-require').install() and then require the template. This isn’t working for me in Marko 5 in a CommonJS context.

But, I am only interested in the Node ESM output/context. markoc is able to compile the .marko templates without any issues, but the ESM output is not compatible with Node’s import requirements (no extensions to marko imports, etc.). I considered stringing Rollup or something else to transpile afterwards, but this started to feel like a lot of overkill for a template.

Additional Info

Your Environment

• node.js 16.5.0
• macOS 11.5.1

Steps to Reproduce

Clone and run the sample repo: https://github.com/adamchal/marko-5-esm-issue

[DylanPiercey]

@adamchal I am curious how you plan to do bundling with this setup?

Also for context we've mostly imagined that people should just be using the Marko require hook in order to import Marko files in nodejs rather than the load api. I think your problem would probably be solved as well if we implemented a ESM loader for node js?

[DylanPiercey]

I wrote a really basic ESM loader for Marko (code below), however it seems there would need to be some additional changes to Marko's compiled output to fully resolve some of the imported paths. Currently the output templates rely on nodes module resolution for commonjs.

import { URL, pathToFileURL, fileURLToPath } from "url";
import compiler from "@marko/compiler";
import { cwd } from "process";

const baseURL = pathToFileURL(`${cwd()}/`).href;
const extensionRegex = /\.marko$/;

export function resolve(specifier, context, defaultResolve) {
  const { parentURL = baseURL } = context;
  return extensionRegex.test(specifier)
    ? {
        url: new URL(specifier, parentURL).href,
      }
    : defaultResolve(specifier, context, defaultResolve);
}

export function getFormat(url, context, defaultGetFormat) {
  return extensionRegex.test(url)
    ? {
        format: "module",
      }
    : defaultGetFormat(url, context, defaultGetFormat);
}

export function transformSource(source, context, defaultTransformSource) {
  const { url } = context;

  return extensionRegex.test(url)
    ? {
        source: compiler.compileSync(source, fileURLToPath(url), {
          sourceMaps: "inline"
        }).code,
      }
    : defaultTransformSource(source, context, defaultTransformSource);
}

[adamchal]

For a quick background, we have a significant amount of work that constitutes simple landing pages that do not maintain state or require UI. We started with some static site generators and over the years developed our own. We soon switched from Pug to Marko, because at the time it was the only template with support for async data and could generate static HTML without any undesired JS overhead in the output. Even still today, the only other one I have found is Eta.

We also work on a lot of Next.js/Nuxt.js projects of course, but we try not to introduce these UI frameworks to places they don’t belong. We have also dabbled with Svelte and we have high hopes for Sveltekit and its ability to prerender all of the frontend JS away on some/all pages. It would be great if we could use the same framework to build complex UI+state application and simple landing pages.

For our Marko use case, we currently use v4 in a CJS context with the marko/node-require loading approach. This does not appear to work the same way in Marko 5, even in the CJS context. What we’re looking for is a reliable way to compile our Marko templates in Node.js (ideally the ESM context), pass in a data object (full of Promises), and render the HTML output to a file.

@DylanPiercey thank you for the quick reply and sparking this solution! Ha, I didn’t even consider an ESM loader. The ESM loader has been such a painful and controversial topic, that I’ve just blocked it out. That said, we are already using ts-node/esm as a loader. Given the current inability to use more than one ESM loader, this would not work easily for us.

My current ugly solution is to put all of the marko templates in an array, loop over them and marko.load() each one. If it doesn’t throw, remove it from the array. Continue looping until the array is empty. Disclaimer: I thought long about whether or not I should share or admit this, but here we are. :)

This is not a reasonable solution, but since we do not need or use the intermediate compiled .marko.js files, the approach solves our problem.

If we pursue an ESM loader approach, we would need to markoc or somehow otherwise compile the templates, then read them with the loader. This would still be somewhat clunky, even if Node.js supports multiple ESM loaders in the near future. A similar approach would be to tool rollup or babel to convert the Marko templates into standalone modules that can be imported.

The biggest problem with these approaches is that invalidation becomes very difficult—especially in an ESM context. This is another ESM pain point, but the import cache workarounds are tedious.

In a perfect world, we would want to be able to load, compile, and render the .marko templates directly in Node.js without having to worry about the intermediate .marko.js files and the limitations of the CJS/ESM loaders.

[DylanPiercey]

@adamchal I know this isn't an ideal solve, but just stumbled upon https://github.com/node-loader/node-loader-core which is a utility to merge node loaders so you can run multiple.

[adamchal]

@DylanPiercey interesting find and might be workable—thanks!