Can 11ty be instructed to repeatedly poll external API data while running in serve or watch mode?

[kkgthb]

I'm working on a proof of concept where I host an Eleventy project running in "--start" mode on Heroku (the only general-purpose long-running NodeJS host I have any familiarity with), intended to function as a "live preview" playground for content authors working in Sanity CMS (or any other API-based CMS).

I can see a few blog posts and starters for Contentful, DatoCMS, Sanity CMS, etc. but before I get too deep into the weeds, a question I can't quite find an answer to myself is:

Can you get 11ty to "watch" for external data changes?

I see that a server running 11ty in "--serve" mode will watch for changes to files in its local filesystem, but since a host like Heroku will restart the server every time I use Git to make a change to the files in its filesystem, that's not exactly useful.

That's what took me back to the idea of API-based CMSes.

Gatsby has a gatsby-source-sanity plugin that can be configured to frequently ping Sanity's "drafts" API options such as watchMode and overlayDrafts. To me, this implies that Gatsby SSG itself can be told to frequently poll an external site while running in develop mode.

Is there a way to tell Eleventy to "frequently poll an external site while running in serve or watch mode?" I'm struggling to find documentation or tutorials.

Or is that just something that Eleventy can't do? (Don't want to waste my time to find this out the hard way!)

Thanks so much.

[siakaramalegos]

I don't know if this is helpful, but I have what amounts to a fake cron job through GitHub actions that sends a webhook to Netlify to rebuild my website every few hours to bring in new webmentions. It's still completely static, but it brings in new information several times a day. I don't know how frequent you could go before hitting the max of the free tier.

[binyamin]

I don't know how frequent you could go before hitting the max of the free tier.

@siakaramalegos I actually worked it out once. You can go much more than a couple times a day, and stay within the free tier.

I'm working on a proof of concept where I host an Eleventy project running in "--start" mode on Heroku

@kkgthb I don't think Eleventy will help you here. Eleventy is simply a tool to generate static sites. Static sites don't generally change after they are generated. It's completely possible that you can write the website's JavaScript to accomplish what you want, but Eleventy won't help this.

Also, eleventy --start only adds a web server, which is exactly like running eleventy by itself, and uploading the output to Netlify.

[kkgthb]

@b3u I thought that eleventy --start adds a web server that dynamically watches for changes to its filesystem and rebuilds individual pages in response to those changes, and doesn't just run a static-for-the-whole-time-it-runs web server?

If so, that "watching and responding to triggers" functionality is essentially what I'm asking if I can tap into for other changes, like API endpoint changes. (I do like your point that firing the trigger would potentially have to be done by a client-side API listener, though.)

[binyamin]

@kkgthb Just fyi, I checked the documentation and there is no command eleventy --start. You probably mean eleventy --serve, which does indeed watch the filesystem for changes. the command eleventy, without arguments, just starts a web server.

To answer your question, I don't know how to do that with 11ty. I imagine it could work, for example, if you customized the browserSync instance. However, I'm sure there are many tools, methods, and workflows which would do the job more easily. Again, that's not what it's built for. I can poke around.

[kkgthb]

Thanks @b3u -- that was indeed a typo. eleventy --serve! :)

I think I found another way to hack it. Just built out a proof-of-concept this morning.

https://github.com/joerobot/sanity-11ty gave me some boilerplate for making --serve builds "prefer" Sanity "drafts API" content over "published API" content at build-time.

Then borrowing from https://medium.com/@richyinbeta/eleventy-strapi-headless-cms-with-automatic-page-creating-and-updating-86942f10c19a, I set up an Express server on a different port with an endpoint reload that writes the sysdate to a file '../web/dummyFileToForceReload.njk' and run it in parallel alongside eleventy --serve.

Now I can force a refresh of https://localhost:8080 files by visiting https://localhost:3005/reload.

Hopefully I will get this written up & put on the web sooner rather than later.

[binyamin]

@kkgthb that's great to hear. If this solution works, would you please close the issue?

[kkgthb]

Sure thing, @b3u.

One final note to self: if playing with "plugin" building ...

https://github.com/sanity-io/gatsby-source-sanity/blob/main/src/util/getDocumentStream.ts & sanity-io/gatsby-source-sanity@ad7a57b#diff-58a996f947cfabb0eb6cd1fcb7e09e62 (its first commit) are examples of how the gatsby-sanity-source NPM package leverages other NPM packages made for having the running Node server "subscribe" to streaming events.

See also https://github.com/sanity-io/gatsby-source-sanity/blob/main/src/gatsby-node.ts and search in-page for the word "overlay" and "watch." It seems to make use of NPM's @sanity/client's .listen() (https://github.com/sanity-io/sanity/blob/next/packages/%40sanity/client/src/data/listen.js).

Because it does so within the context of a sourceNodes export in a gatsby-Node file, this "running stream" is, I think, a little bit "magic" from the perspective of a Gatsby server in "develop" mode and Gatsby handles on-screen rendering from there. (All the plugin has to do is keep altering its sourceNodes export according to the stream it's subscribed to.)

As @b3u mentioned, because 11ty URLs are just HTML files with no inherent "phone home to the server's running NodeJS code" JavaScript in their <script/> tags, getting "DOM updates" from the @sanity/client`'s stream of "new info" to a viewer's web browser will require building new DOM content somehow and using BrowserSync somehow to force it through to the viewer's web browser.

In theory, something like browserSync.stream() might be able to cause the "as-you-type" effect (https://stackoverflow.com/a/46787136), but in practice, without that full "secondary DOM" that Gatsby injects as JavaScript and without all the built-in server-to-client rerendering of "nodes" that Gatsby includes out of the box, it could be very annoying to try to manipulate the contents of the page "live" rather than just deciding to have an end user say, "I'm ready to see my work now" and do a full browserSync.reload(). (browserSync.reload() comes "out of the box" in 11ty if you alter its filesystem and trigger a rebuild.)

At some point, without triggering full rebuilds, with the amount of trickery to browserSync.stream() and injection of JS into the HTML so as to make the DOM of every page depending on some source data, one is just rebuilding Gatsby inside an 11ty plugin. As noted at #108, a "page" itself isn't necessarily the only HTML file that needs to look different upon source content changes. If one changes the title of a blog post, the blog listing also needs to reflect the changes.

So we're back to full-rebuild magic links. You might as well just have 11ty templates inject a "refresh me from drafts" button into the corner of every page in development builds. Thankfully, 11ty seems to rebuild fast.

This means that perhaps the only reasonable architectural optimization left, as far as I can see it, would be the possibility of using "plugins" as a way of slipping a little more "web server" functionality into localhost:8080 to sneakily move the "magic endpoint" inside localhost:8080 instead of running an express server on localhost:xxxx. I suppose I'll have to poke around @11ty/eleventy itself to see if this is even possible.

Note to self: If running preview servers in eleventy --serve mode in the cloud, don't forget to password-protect the site -- at least the "magic endpoints." Don't want to give people a great time DDOSing my Sanity API or Heroku dyno.

One more note to self: since all the thing running in --serve mode does is do fast rebuilds (no need to spin up a new server) ... if you can find a "builder" service that builds from scratch in comparable time (e.g. maybe a secondary Netlify with its environment variables set so that it builds from drafts), and if you can figure out an identity-protected way to let a content author hit its "build" webhook as a "magic link" -- e.g. maybe there's a button that does a POST thrown into the corner of every page on this "dev" build and the whole site is behind an authorization wall -- then there isn't actually really a need to do the whole Heroku thing at all.

The builder's webhook system replaces the Express server and we're back to the "rebuild from scratch" architecture I took my inspiration from, only with less nonsense to maintain yourself.

And if Sanity ever adds a sort of authorial "see my draft work" webhook (rather than just "publish" webhooks), you wouldn't even need a magic button in the preview UI anymore. They did say that better webhooks were on the way.

[antgel]

I think #3085 and 11ty/eleventy-dev-server#70 should help with this, if anyone is reading and interested.