Database

[imdfl]

We want to support client side searches. Examples:

• Find all pages that match a label set
• Find all pages relevant to a time range

I think that this functionality can be fully implemented as a client side database.
The flow doesn't even need a cms:

1. All the metadata is stored in the md files.
2. Each build starts by indexing the content, adding everything to an in-memory database instance, exporting the database to a file and optionally committing the latter.
3. The client pulls the database dump after loading and initializes a client-side db (same as in the prev step) with it.

I think that RxDB is a good candidate. It can run in-memory, supports both node.js and browsers and offers export and import of JSON data.

Full text search

I don't have a favorite solution yet. A straightforward client-side solution - loading the full text index on the client side - may turn out to not be scalable.

[tomerlichtash]

My only concern is that client-only approach would make the build process very slow at scale, as well as the MD file base large and atomized (and hard to manage).

If we're going down that road, we have to see that this approach accepts different sources (from different repos), and that there's an acceptable caching solution to the existing schema.

Anyway, if we're going with MD, I think a prior task would be extracting the content from the client, so not to maintain a dependency between content changes, client build, and eventually content caching.

[imdfl]

My only concern is that client-only approach would make the build process very slow at scale, as well as the MD file base large and atomized (and hard to manage).

If you mean content files, we run a full scan of them in every build, so where's the delay? If you mean media files, they don't have to be backed up by md, this is obviously wrong.

If we're going down that road, we have to see that this approach accepts different sources (from different repos), and that there's an acceptable caching solution to the existing schema.

Can you list a use case?

Anyway, if we're going with MD, I think a prior task would be extracting the content from the client, so not to maintain a dependency between content changes, client build, and eventually content caching.

I don't understand why this separation is a prerequisite. The current impl assumes that the content is available under /public/content. This assumption can be met by including the content in the repo, or by cloning/updating it in build time. Perhaps the main obstacle here is our dependency on Vercel deployment. If we rolled our own server, it would be trivial to deploy a client that pulls its dynamic data from whereever. This would entail no architecture or data-flow changes, other than setting up the content in the public folder.

[imdfl]

So we've agreed that the build process will include indexing the content for labels and time.
I want to decouple the indexing from the next build process (NBP), but there's a snag: Iterating on all the content may pull in files that aren't part of the site. Also, it loses the locale data (which we will have to infer from the file name, ugly).

Conclusion: The NBP is the only reliable source for a full list of the pages.

We can dump all the file names to a file during the NBP, but next may build the pages concurrently; Therefore, writing to a common file may result in data loss/corruption, unless we roll our own locking mechanism. I don't want to rely on in-memory data, because there's no guarantee that next will build all the pages in the same process.

Conclusion: we need to run a database app before the NBP, send it all the file names + locales during the NBP and finally run an indexer app (using all the utils we already have) that indexes all the files listed in the database.

[imdfl]

TODO

• optional initial data in database (at least a collection of labels)
• Return full text as string , so we can search it for predefined labels
• Add optional data size limit to response handling in http requests