Improve the developer experience for theme pack / context plugins

[thescientist13]

Currently the workflow for developing and publishing a theme pack is a little clunky, so would be good to track and discuss ways to make this better.

### Development

The main consideration needed for development is that your files won't be in _node_modules, which is what the case would be for users.  So for that reason, we need to add a little biolerplate to _my-theme-pack.js_.  There might be others way to solve it, but it just checks if the packages is installed:
1. If it _is_ installed, then use `__dirname` (which would be inside _node_modules_) as the base path
1. If not installed (like for local development) then use whatever location you have defined in your local repostory.


So using our current example, our final _my-theme-pack.js_ would look like this:
```js
const os = require('os');
const path = require('path');
const packageJson = require('./package.json');
const { spawnSync } = require('child_process');

module.exports = () => [{
  type: 'context',
  name: 'my-theme-pack:context',
  provider: () => {
    const { name } = packageJson;
    const baseDistDir = `node_modules/${name}/dist`;
    const command = os.platform() === 'win32' ? 'npm.cmd' : 'npm';
    const ls = spawnSync(command, ['ls', name]);
    
    const isInstalled = ls.stdout.toString().indexOf('(empty)') < 0;
    const templateLocation = isInstalled
      ? path.join(__dirname, `${baseDistDir}/templates`)
      : path.join(process.cwd(), 'src/templates');

    return {
      templates: [
        templateLocation
      ]
    };
  }
}];
```

And our final _greenwood.config.js_ would look like this
```js
// shared from another test
const myThemePackPlugin = require('./my-theme-pack');
const packageName = require('./package.json').name;
const path = require('path');
const { ResourceInterface } = require('@greenwood/cli/src/lib/resource-interface');

class MyThemePackDevelopmentResource extends ResourceInterface {
  constructor(compilation, options) {
    super(compilation, options);
    this.extensions = ['*'];
  }

  async shouldResolve(url) {
    return Promise.resolve(url.indexOf(`/node_modules/${packageName}/`) >= 0);
  }

  async resolve(url) {
    return Promise.resolve(url.replace(`/node_modules/${packageName}/dist/`, path.join(process.cwd(), '/src/')));
  }
}

module.exports = {
  plugins: [
    ...myThemePackPlugin(),
    {
      type: 'resource',
      name: 'my-theme-pack:resource',
      provider: (compilation, options) => new MyThemePackDevelopmentResource(compilation, options)
    }
  ]
};
```

> _It is a little clunky, so please follow [this discussion](https://github.com/ProjectEvergreen/greenwood/discussions/682) for ways we can try and make this more elegant!_  🙏🏻

[thescientist13]

Yeah, although I wish we could have supported relative paths, if you keep going down, there were some issues 😞
#689 (comment)

If we could make it work though, that would be amazing. Maybe something #691 could help with? 🤔

[thescientist13]

One thing about the isDevelopment flag is that Greenwood will always look in the src/ directory, even when building.

const packageJson = require('./package.json');
const path = require('path');

module.exports = (options = {}) => [{
  type: 'context',
  name: `${packageJson.name}:context`,
  provider: (compilation) => {
    const templateLocation = options.__isDevelopment
      ? path.join(compilation.context.userWorkspace, 'layouts')
      : path.join(__dirname, 'dist/layouts');

    return {
      templates: [
        templateLocation
      ]
    };
  }
}];

Ideally when creating a theme pack and running greenwood build|serve those calls would go to node_modules too.

[thescientist13]

Another observation, when creating the one off development resource there seems to be a discrepancy in how process.env.__GWD_COMMAND__ is getting set, I guess?

process.env.__GWD_COMMAND__ = command;

In the tests I added here I was able to test the command in the resource like this

return Promise.resolve(process.env.__GWD_COMMAND__ === 'develop' ... );

But in my main theme pack repo, I had to add a space?

return Promise.resolve((process.env.__GWD_COMMAND__ === ' develop') ... );

Maybe it's something about how commands are getting passed between npm / yarn in npm scripts? Left the docs to match what I have in my repo just to be safe? 🤷‍♂️

[thescientist13]

Noticed there's also an opportunity to clean up some code added here too as it seems that when theme pack was added and rebased into #674, require.resolve was unable to find the mock "my-theme-pack" and so was failing with an exception

Error: Cannot find module 'my-theme-pack'
Require stack:
- /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/plugins/resource/plugin-node-modules.js
- /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/config.js
- /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/compile.js
- /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/commands/build.js
- /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.resolve (internal/modules/cjs/helpers.js:94:19)
    at NodeModulesResource.resolve (/Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/plugins/resource/plugin-node-modules.js:29:85)
    at /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/serve.js:4:421
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/serve.js:4:216
    at async /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/serve.js:4:216
    at async /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/serve.js:4:216
    at async /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/serve.js:4:216
    at async /Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/serve.js:4:216 {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/plugins/resource/plugin-node-modules.js',
    '/Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/config.js',
    '/Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/lifecycles/compile.js',
    '/Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/commands/build.js',
    '/Users/owenbuckley/Workspace/project-evergreen/repos/greenwood/packages/cli/src/index.js'
  ]
}

So had to add a fallback in that case, to what our prior logic was, just to be safe.

---

Would be nice to find a way to make sure this can interop nicely with our (Theme Pack) tests so we can remove this, although I suppose leaving the error handling in is fine.

[thescientist13]

Good news is that with #684 , these annoying console logs are now gone - thescientist13/greenwood-starter-presentation#73 🎉

[thescientist13]

Another opportunity to improve things for Greenwood, not so much developers, is in order to support #674 , we needed to add a work around for Theme Packs specifically. Maybe there could be a way to eliminate this, possibly in conjunction with #691 ?

[thescientist13]

Even after the v0.27.0 upgrade, for theme packs it seems, like the [greenwood-starter-presentation[(https://github.com/thescientist13/greenwood-starter-presentation) for some reason aren't bundling <style> tags in markdown files?

thescientist13/greenwood-starter-presentation#63

[lschierer]

After getting my personal theme pack working I have a couple of thoughts.

The biggest is that it would be awesome if the Context plugin allowed me to implement the same object as Compilation.context, and have the "default" context from the consuming project's workspace get merged with the one provided by the plugin. I recognize that isn't a small ask.

More minor, but potentially just as difficult to implement, is that on a number of occasions I hit situations where it was difficult to figure out what I was doing wrong because I couldn't get console.out anywhere in my code to work. It would really help if I could turn on, ideally something on the command line I could put in my scripts section of package.json, the ability to trace as plugins get initialized so that I can see

1. what order my plugins are executing in so that I can debug interactions between them
2. any errors that are occurring when Greenwood attempts to load them.

For this second one, I want to emphasize that each time this happened, when I did trace it down, it was always in fact something that I was doing wrong in my code. It just would have helped to have a bit more in the way of guardrails to tell me what I was doing wrong.

[lschierer]

I didn't try editing Greenwood itself, I mean that a console.log statement in the plugin registration of any plugin wasn't firing even though it was clearly delaying performance potentially infinitely though I get impatient and kill things, because removing it would allow things to work.

[thescientist13]

Could you maybe perhaps provide a concrete example perhaps? Are you saying something like this wouldn't produce any logging output for you?

export function myContextPlugin() {
  return {
    type: "context",
    name: "acme-theme-pack:context",
    provider: () => {
      console.log('Running provider function of my context plugin!')
      
      // ...
    },
  };
}

[lschierer]

it will when everything is working. It won't under certain poorly understood (by me) error conditions.

[lschierer]

More,

export function myContextPlugin() {
console.log('Running provider function of my context plugin!')
  return {
    type: "context",
    name: "acme-theme-pack:context",
    provider: () => {
     
      // ...
    },
  };
}

this won't either under the same conditions.

[thescientist13]

I just tried that here and it seemed to work for me?
https://github.com/thescientist13/greenwood-native-typescript/blob/unified-in-plugins/external-content-source-plugin.ts#L19

Do you have an example repo somewhere?

[lschierer]

I don't want to configure any of them. It occurred to me that a read only view of them would let me know anything the greenwood.config.ts file knows. I could get reusable values for plugins like site title similar to the way you can with plugins in vite for essentially free (to me as a plugin author). It lets a plugin more easily allow for configured or rather parameterized behavior, which has a number of use cases for a resource plugin.

[thescientist13]

Could you provide a code snippet and / or link to the Vite docs of what you mean? Just want to make sure i understand, because plugins all get a copy of the compilation from the provider functions, which you can then access context from.
The same goes for config as well.

https://greenwoodjs.dev/docs/reference/appendix/#compilation

[lschierer]

before I found Greenwood, I looked at Gracile but it was, at least at that time, at a very early stage of its lifespan with its sole developer at that time able to spend very little time on it. And I needed more experience to understand any number of things that would have made me successful on Gracile as well. Anyway, one of the things I saw with Gracile but never fully understood is that it is entirely based on being itself a Vite plugin. It is entirely configured from vite's configuration file.¹ I never fully understood this part, but one of the things that Gracile offers is that you can theoretically consume any of the vite ecosystem plugins. Gracile is getting things like its own hostname and port number from vite, it isn't being configured in some Gracile-specific configuration object. It was in this context that I was thinking about Theme Packs and their related plugins.

A Plugin that provides a layout might well want to know the site name as a variable from the Theme Pack that provides it. But I don't want to hard code that somewhere, then the plugin is specific to that project. It needs to be a variable. The natural place for this is the greenwood config file. I eventually figured that out, but only by passing all of the theme pack's variables in as parameters to the theme pack. I've seen that pattern with Rollup and postcss, where you see plugin[] blocks that are frequently a mishmash of configuration for each of the various plugins in that array.

Before any of this rather wild and crazy journey, I come from having using Astro and one of the things I eventually realized as I explored migrating away from it is that Astro's Configuration is in fact the Vite configuration objected extended, or rather, wrapped. I can put any vite configuration option in the 'vite' section of the astro config because that portion of it is a type-safe vite config object - just as you would expect when you wrap one object with another. Then when I use Astro's Starlight template, I get a plugin that, though their documentation doesn't fully show it well, fully reads the parent Astro config. Here is the production config file for the HP_Stuff project you've helped me with more than once, since I don't have a deployable working version to migrate to yet. Starlight is in fact using that both the 'site' and 'trailingSlash' options that the parent astro defines for its layouts and its routing behavior - as the Starlight plugin affects how Astro does both (its a massively intrusive plugin that's great when you aren't ready for all the decisions that I've had to make as I've left both it and Astro behind). You'll note this is behaving very similarly to how Gracile does getting configuration from the vite.config.ts file - the Gracile documentation is clear that there's been some inspiration happening.

I guess I was hoping that since Greenwood doesn't strongly type its configuration object that it would import it extra keys and all, that I could define a site title, and other global variables that multiple plugins need at a top level rather than deeply nested more the way Rollup does it. then a plugin can say if ('mykey' in GlobalConfigObject) { .... }

Footnotes

1. https://gracile.js.org/docs/learn/getting-started/configuration/ ↩

[lschierer]

shudders I once saw a postcss config file that had plugins consuming plugins, at least one of which took a (shudders again) function as a configuration option. I never did come to understand anything of that config file example. Theme packs feel almost like that to me and even having mostly gotten one working, part of feels like its black magic that I lucked my way through ;-)

[thescientist13]

fair, and so I appreciate any and all feedback provided here 👍

[lschierer]

A read/write use case is that a plugin could inject its own directory into any of the search paths as it already does for layouts. Off hand that might only be useful for a context plugin paired with an external source plugin. The read only use cases are, I admit, at the forefront of my thinking.

[thescientist13]

So just layouts and pages? Or more than that?

As I mentioned for layouts, Greenwood has to make sure it handles all paths in all places that may use layouts (including default location), and so the same would go for pages (and I guess implicitly API routes 🤔 ). I think those two could probably supported (and I have that other issue for handling pages), but if you can provide examples of other context properties you would want to have userland provide to Greenwood and the things it would help you achieve, that would be helpful context and help me better understand what kind of things folks are trying to build with these plugins and Greenwood, just to make sure I have a wholistic view of things.

[lschierer]

I thought about plugins and components, but ultimately I don't know how much it would help. the major pain point with both of those is that setting up the package.json and the build output in the dist directory are both insanely fragile, not intuitive, and none of that is Greenwood fault, nor fixable by Greenwood. I'm not sure there is anything the theme pack could do about either because at the end of the day Greenwood has to be able to do dynamic imports on those bits of code not just during the compile phase, but the resulting javascript produced by the compile has to work as well.

that's different from layouts where while you still import them during compile, they produce a static output file that, once its in the consuming project's public directory neither knows nor cares if it came from a layout.html or a layout.ts or if that layout was from the consuming project or a plugin. That seems like something that might be (incredibly) hard to fix, but once fixed, is fixed for everyone always and the bar to attempting a basic theme goes way down.