implement backends interface and nats afero.FS and nats watcher.

[nickchomey]

Closes #63

This PR adds a backends interface that allows for using different afero.FS backends along with providing a custom Watcher to allow for hot reloads. It also implements a NATS-based afero.FS and Watcher.

Made changes to allow starting xtemplate without any templates loaded, so that templates can be added after startup.

It comes with an "app-nats" test app that is run by cue cmd build_test_nats, which populates the nats object store with templates after it has already initialized, triggering a hot reload. Then it runs the hurl test suite against it. The other tests (or at least build_test_cli) might want to be modified to test the watcher and reload as well. I assume you could polish off the CUE stuff - I dont have a clue about how it truly works.

Also fixed a bunch of small lint errors (sorry, i should have done that in a separate commit at the very least, if not PR...)

It is important to note that I built this on top of your WIP next branch, and am merging it there. If you want to wait til you merge that to main, that is fine.

I also added a mise.toml file, which allows for easily installing all of the required tooling (caddy, xcaddy, nats, etc..). Mise is absolutely beautiful, and the future of such tooling as far as I'm concerned.

If there's any changes that you dont understand or have suggestions for changes, please let me know! I could retroactively create various commits, or even split to different PRs if that would help you to review this.

[Copilot]

Pull Request Overview

This PR implements a backends interface that allows different afero.FS backends and custom watchers for hot reloads, along with a NATS-based implementation. It also adds the ability to start xtemplate without templates loaded initially, enabling templates to be added after startup.

• Introduces a new backends interface with filesystem and NATS Object Store implementations
• Adds support for starting servers without initial templates (graceful degradation)
• Updates the provider initialization system to support backend creation

Reviewed Changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
config.go	Adds Backend field and duplicates CrossOrigin field
backends/	New backend interface and implementations for filesystem and NATS Object Store
server.go	Updates server creation to handle missing templates gracefully and backend management
instance.go	Refactors provider initialization order and backend setup
app/app.go	Updates watcher setup to use backend-provided watchers
test/	Adds NATS test application and configuration

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[nickchomey]

uh oh, i rebased on top of your most recent few commits just before submitting - it seems like there's some compiler errors. I'll push a new one soon

[nickchomey]

Ok, i resolved a few merge conflicts, as well as added arg:"-" to the CrossOrigin field in the Config struct - it was preventing the app from running.

cue cmd ci passes for me. Hopefully the automated tests here will do the same

[nickchomey]

looks like the check failed because it cant log into docker hub

[infogulch]

Thanks for opening this, Nick! I think this could be an interesting capability to have in xtemplate.

Scanning through the diffs it looks pretty good so far. To share my first live reaction, I'm a bit hesitant about adjusting the order of operations: initializing the providers first, which can modify the backend state, how that affects reloads etc. I don't see any obvious problems yet, but this stuff is tricky so want to consider the implications.

Don't worry about CI, getting CI to work with contributor PRs requires reps to iron out the kinks, I'll look into it. I will pull your branch to review more carefully and test locally this week anyway.

[nickchomey]

Yeah, it's all tricky for sure. I really did my best to not make any changes at all to xtemplate, and in fact there initially weren't any meaningful changes. But as I tested things I kept finding edge cases and thats where things like the re-ordering, adding pointer to nats config, adding config param to dot providers etc came in.

I also initially had a WithBackend method to add a backend, but then opted to bake it into WithNats given that it's somewhat inseparable from the nats provider, given that they'd want/need to use the same nats server.

But it just occurred to me that we probably would want to add back the WithBackend for use with other potential backends (s3, git etc) that perhaps don't rely on a provider. I'll push a commit to add it back today.

It is entirely possible that there's other, better ways to do it all, but I think I'm tapped out on insights (and energy). You know the codebase and vision far better than I do, so if you can think of a better way to do anything, I'd be happy to make changes.

It's not urgent that this be merged - I'm going to focus on some other stuff for a little while. So, get to it whenever you have time!

[infogulch]

Nice, CI passes now. 🎉

I took the liberty to pull out some pieces of this PR into the next branch while preserving your authorship, which leaves this PR to be more focused on the backends feature. I hope you don't mind. :)

• mise is very nice, great find 💯 using it to install tools in CI now too
• fix lints 👍
• CORS build failure fix 😅

Some other changes: fixed CI for external contributors, and adjusted recent commit messages on the next branch and on this PR.

As for what remains:

• Being able to finish starting the server before the template files are ready and serving a 503 instead of exiting immediately is a good idea. This will be helpful for remote backends whose deployment may be somewhat desynced from the server start.
• I'm pretty sure there's a simpler strategy for nats and backends floating around nearby. I'd like to refactor this and open a separate PR so we can compare and contrast approaches. Here's what I'm thinking at the moment:
  • Drop the backends abstraction, move the Watch responsibility to the app layer. can keep the watch backend code, just moving it
  • Reconsider if the Reload API is sufficient. E.g. perhaps add some kind of optional signaling mechanism to the Config struct (a channel?) to allow reloads to be triggered more easily
  • Keep the old nats initialization system, lift nats backend server init into the app layer to preserve the nats server across instance reloads. not sure if this will work with config needs
  • The new cue can be factored a bit more
  • Push objects to nats object store with cue-driven cli instead of app-nats which should not have test code

[nickchomey]

ah, glad youre receptive to mise! It's incredible. I originally used the go backend, which just seems to run go install, but recently they added the github backend, which makes it easy to install release artifacts, which is more efficient.

I might as well also mention jujutsu vcs and its jjui TUI - complete "gamechangers" for doing version control. On that note, I hope it wasn't too much effort for you to extract the linting fixes - I could have done it fairly easily if you had asked for it, as jj has an interactive split mode that makes it dead-simple to extract specific lines and sections of code that have changed.

I'm very much an evangelist for jj, jjui and mise, spreading the Good Word wherever I can.

---

for the remaining tasks, I'm completely open to any refactoring that you want to do - I was trying to be as non-invasive as I could, but if you're willing to change xtemplate in various ways, im sure that there's better, simpler, cleaner ways to implement all of this. I'm happy to receive suggestions, but its probably best if you do it as you understand the codebase, vision, what you want to maintain etc... I'd like to think that I've at least done the heavy lifting of implementing the NATS afero.FS and watcher, and showing a working PoC of how to use them with interfaces.

• Backends: I'm not sure how it would work if you remove the backends abstraction. How would we pass in/use different afero.FS and associated watchers? Or are you saying to just decouple the afero.FS and Watcher interfaces, rather than have them both in the same Backends abstraction?

• Reload API: Yes, this sounds good to me. I also don't love how the whole thing rebuilds any time the watcher detects a change. Something more incremental seems like it would be ideal. But, perhaps you had good reasons for doing a full reload and changeover?

• NATS Server Init; Yes, I'm not sure where NATS server init would be best done. dot_nats_config is doing all of the init etc, but perhaps would be best just as pure config. I was also working on the assumption that NATS (and anything beyond just normal FS) was an optional component. But if you've also drunk enough of the NATS koolaid and want to make it a 1st class citizen, that would be more than fine with me.

• CUE tests: I had done a test implementation using nats cli via cue to upload the templates to the object store, but i couldnt figure out how to do simple things in cue, so i ultimately just reverted to the app-nats approach. The idea was not for it to be a real app, but instead just an app meant for testing. So i didn't see much problem with mixing the app and tests. Anyway, i have no objection to you moving the test functionality to cue - in fact, im curious to see how you do it.

• Another idea is Mise + CUE; Mise has a comprehensive task runner, but it seems like cue is more expressive. Perhaps they could be used in conjunction? Maybe use mise as the entry point to the cue tasks, since mise can run tasks in parallel? eg mise test:cli could literally just call cue cmd build_test_cli, and mise run test:* would run test:cli, test:caddy, test:nats etc... at the same time. And the [depends] and [wait_for] can be used in place of $after.

Anyway, glad you're mostly receptive to all of this. I look forward to seeing what it becomes! Let me know if I can help with anything.

[infogulch]

I'm pretty familiar with git so it wasn't too much trouble to shuffle some hunks around (fine I admit AI helped a bit). Everyone raves about jj and I'm sure it's great, but I didn't dedicate enough time to become proficient and screwed up my branches with it. Trying it again is on the list.

decouple the afero.FS and Watcher interfaces

Yes that's the idea.

perhaps you had good reasons for doing a full reload and changeover?

xtemplate loads all files into a single instance of templates.Template, mutating it as it walks the templates directory tree. Each file can define an arbitrary number of additional templates with arbitrary names which may clobber other definitions or pollute the template listing. I really really want to avoid situations where the state of template definitions loaded on the server depends on the sequence of past deployments; this is how you get nightmare heisenbugs.

It may not look like it, but template definitions are code. Imagine taking a ruby on rails app and checking out a random subset of ruby files from one commit and the rest of the files from the next commit and running it. There is no telling what the result could be, everything could be mostly fine or there could be a site breaking bug or a giant security vulnerability, even if each commit in isolation works perfectly.

I'm doing this full-reload approach because I do not want to play with this kind of issue. Honestly even the current debounce-based reload is risky.

There might be a way to cache intermediate template parse results, so the templates.Template instance can be stitched together without re-reading and re-parsing the files, but I haven't seen a use-case where template parse time is a huge problem yet, and this is already mitigated by by keeping the current instance running and building the new instance in the background then flipping it with a single atomic ptr write once the new instance is ready.

Do you see how being loosey-goosey with template loading could be a huge problem?

NATS Server Init

One could say I've already drunk the coolaid, since nats is embedded and configurable in all xtemplate deployments today. You can either let xtemplate create your nats client and optionally server (per instance because that's the configuration/execution state boundary used by xtemplate) OR you can provide an already-connected client in which case xtemplate will just use it.

If you want a persistent server, instead of pushing creation and management of the server down into the xtemplate module where there's some impedance mismatch with how xtemplate manages instance state, persistent things should be created by the caller (i.e. your own /app or /cmd) and a nats client provided to xtemplate instead.

CUE

Yes CUE cmd is tricky to write, honestly I'm impressed that you got it working! CUE is already very parallel since everything runs in parallel by default unless you explicitly sequence with $after. It's so parallel in fact that I've had to limit parallelism in CI or the tests bottleneck on IO waiting to read the template files and tests time out. Even if I end up pushing the implementation from go into cue cmd it will be very nice to have a working example to compare against.

[nickchomey]

re: full reload. Fair enough! Thanks for the context. It seems sensible as-is.

yes, i agree that a persistent nats server - embedded or otherwise - would make more sense in /app or /cmd. I actually first implemented it like that and passed it through, but then I found dot_nats_config and figured I'd try to integrate it there.

In fact, it seems to me that my first cut with everything was actually the cleanest - i only started getting invasive when I started discovering other mechanisms in xtemplate and trying to integrate with them. I dare not try to dig it out of old commits though - things got messy, despite jj.

If you can think of a way to extract the embedded server from the config to the app/cmd, but in a way that is still easily configurable/importable, that would be great!

Ah ok, I didnt realize (or perhaps remember) that Cue is parallel. Perfect! Mise can just install tools then.

Let me know how i can help with anything!

[infogulch]

Interesting to hear about the history of this patch. Yes, the hardest part of developing xtemplate has been deciding exactly where to cut in features, it has a bunch of layers that are different from typical apps so it's often unclear where features should live until I play with a few different approaches.

If you can think of a way to extract the embedded server from the config to the app/cmd, but in a way that is still easily configurable/importable, that would be great!

🫡 I will try, wish me luck!

We discussed how you'd like xtemplate to enable your desired architecture in #63, but we didn't really get into the how and why, and to be honest I'm still a bit puzzled why you want this feature. Maybe it would be helpful if you expanded a bit on how you hope to use the nats object store backend: What does the site do? Who are the users? Where to do the generated templates come from? Why do they change so often? What do they contain?

To be clear, I'm not trying to qualify the feature, I'm just curious.

[nickchomey]

I'm not at all trying to be evasive, but its hard to describe what exactly I'm trying to do... I won't really be exposing any of this to end users (though, I strongly suspect that @joeblew999 will want to do something like that - we're friends so I'm familiar with what he's up to). But, in essence, I just want to be able to somewhat dynamically generate templates from a UI and/or other programattic mechanisms, as part of a fairly dynamic, reactive web application - they are unlikely to ever be (or at least not primarily) actual files in the filesystem.

My constraints/needs are that I want it to be

• distributed/safe - normal single-node filesystem is insufficient. s3 and git have been discussed in other issues to suit this need, and they'd be fine.
• reactive - I want all nodes to be immediately using the same templates. Im sure this could be done with s3 and git in some way, but it seems to me that it would be considerably more complex than just using NATS which has the reactivity built in, and the Object Store in particular essentially behaves as a distributed filesystem that syncs everywhere in real-time and is also tied right into the application and other pubsub/jetstream stuff. And even if you are only running on one server, there isn't really any cost to it - its just a thin abstraction over the filesystem. Its fsnotify + rclone on steroids, and simpler.

Is this at all helpful? It seems to me that its just a good idea in general, regardless of the weird things im doing in my app.

[joeblew999]

I’m building this thing to keep all files in an S3 and for each serve to pull them to local disk only when they are needed .

so local fs is really a cache .

Using S3 ( I use R2 ) is important for me so that servers stay stateless .

it’s in my repo but not finished.

it’s generic so could be useful with xtemplate.

if anything changes S3 , A simple HTTP hooks fired off R2 so that all nodes know too and will update local cs he is needed.

If you mutate local file , it will make sure S3 ( R2) updates .

So xtemplate is working off Local FS, but files are really on S3