Add PageFind beta V2 (Full Text Search across Site) (#2857)

* Copy gerteck's pagefind implementation

* Add initial agolia styling & remove redundant code

* Implement temporary url fix

* Add arrow key navigation support

* Add auto highlight first result

* Add hover focus support

* Increase results limit to 100

* Support pagefind.json configuration file

* Change logic of how url is fixed

* Remove pagefind.json support & Add site.json support

* Add glob configuration support for pagefind in site.json

This added support for configuring Pagefind search indexing via
site.json is aimed towards allowing users to declare glob patterns to
customize what pages in which directory should be indexed by pagefind.

* Add support for multiple glob patterns

* Update userguide and remove incomplete configuration options

Initially, I felt that including the root_selector option and
force_language option would be useful for pagefind configuration
options. But perhaps for now it isn't the most pressing issue so I'm
going to exlcude it from the current implementation of pagefind.

* Update styling to be more like cards

* Remove pagefind configuration from site.json in docs

* Add error handling for indexSiteWithPagefind()

This helps resolve an issue with generate() test case within
Site.functional.test.ts. Namely, because Jest's cannot intercept
eval('import("pagefind")') runtime dynamic import used by pagefind.

Hence, for now at least, the error handling would catch this error and
allow the site generation to continue normally.

* Resolve lint errors

* Update tests

* Add tests for SiteGenerationManager

* Create Search.spec.js tests

* Fix minor lint issue

* Exclude pagefind generated files from file count & comparison

* Ignore pagefind directory

* Update directory to ignore

* Code quality improvements for Search.vue

Extracted processResult into its own helper function.
Added rel="noopener noreferrer" attribute for improved security and
privacy when being redirected to pagefind's github repo.

* Add validation for glob patterns

* Allow absolute path as valid

* Fix search icon svg issue

* Fix lint issue

* Resolve issues raised in Search.vue

* Update pagefindCss/Js to injected only if index succeeds

Now indexSiteWithPagefind returns a boolean which is then determines if
there is a need to inject the pagefindcss & pagefindJs.

* Update documentation

* Add support for windows glob pattern

Since isValidGlobPattern is only used by normalizeGlobPattern, changed
it to private access. Now normalization of pattern happens solely in
normalizeGlobPattern and validation is purely a validation method.
Better separation of concerns.

Also resolved a bug where the 404s required pages to be created before
knowing if indexing succeeds by defaulting index success to true and
only if it fails does the flag get set to false.

* Fix typo in search.css

* Update docs

* Resolve merge conflicts

* Use static import for pagefind

* Remove uncessary methods

Author: MoshiMoshiMochi

.eslintignore

@@ -6,6 +6,8 @@ node_modules
 
 packages/cli/src/lib/live-server/*
 
+packages/cli/test/**/pagefind/**/*.js
+
 # --- packages/core ---
 
 # Ignore JS files that are compiled from TS

.eslintrc.js

@@ -2,6 +2,7 @@
 /* eslint quotes: ["error", "double"] */
 
 module.exports = {
+  "ignorePatterns": ["docs/_site/**", "**/dist/**", "**/node_modules/**"],
   "env": {
     "node": true,
     "es6": true,

.gitignore

@@ -46,6 +46,9 @@ packages/core/template/*/_site
 # Generated site (MarkBind)
 packages/cli/test/functional/*/_site
 
+# Generated pagefind directories for sites and in expected
+packages/cli/test/functional/**/pagefind
+
 # Ignore .page-vue-render.js files in functional test and subdirectories on update
 packages/cli/test/functional/**/*.page-vue-render.js
 
@@ -117,6 +120,9 @@ packages/core/src/lib/markdown-it/patches/**/*.js
 .nx/cache
 .nx/workspace-data
 
+# Pagefind fragments
+*.pf_fragment
+
 # AI Tools directories (symlinks)
 .opencode
 .cline

.stylelintrc.js

@@ -6,5 +6,16 @@ module.exports = {
     // MarkBind generates some blank CSS files when initialising a site,
     // which violates the no-empty-source rule
     "no-empty-source": null
-  }
+  },
+  "overrides": [
+    {
+      // pagefind uses BEM-style class names (e.g., .pagefind-ui__result) as default.
+      // Since we currently style pagefind's default UI classes, we need to ignore the kebab-case rule here. 
+      // This override should be removed once we no longer rely on pagefind's default CSS classes.
+      "files": ["**/pagefindSearchBar/**"],
+      "rules": {
+        "selector-class-pattern": null
+      }
+    }
+  ]
 };

docs/userGuide/makingTheSiteSearchable.md

@@ -37,6 +37,112 @@ You can add a search bar component to your website to allow users to search the
 <include src="syntax/keywords.md" />
 <include src="syntax/indexing.md" />
 
+
+
+
+
+## Using Pagefind (Beta)
+
+
+MarkBind now supports [Pagefind](https://pagefind.app/), a static low-bandwidth search library, as a built-in feature. This provides full-text search capabilities without external services.
+
+<box type="info">
+This is a <strong>beta</strong> feature and will be refined in future updates. To use it, you must have <code>enableSearch: true</code> in your <code>site.json</code> (this is the default).
+</box>
+
+<box type="warning">
+The Pagefind index is currently only generated during a full site build (e.g., <code>markbind build</code>). It will <strong>not</strong> repeatedly update during live reload (<code>markbind serve</code>) when you modify pages. You must restart the server (re-run <code>markbind serve</code>) or rebuild to refresh the search index.
+</box>
+
+To add the Pagefind search bar to your page, simply insert the following element where you want it to appear:
+
+```md
+<search />
+```
+
+The following UI will be rendered, which is provided by Pagefind:
+
+<search />
+
+<br>
+
+### Ignoring Individual Elements from Pagefind Search
+
+You can exclude specific elements from the search index by adding the `data-pagefind-ignore` attribute to them:
+
+```html
+<div>
+    <h1>This content will be in your search index.</h1>
+    <div data-pagefind-ignore>
+        This content and all its children will be excluded from search.
+    </div>
+</div>
+```
+
+For more details, see the [Pagefind documentation on removing individual elements](https://pagefind.app/docs/indexing/#removing-individual-elements-from-the-index).
+
+### Using Pagefind Configuration
+
+You can customize Pagefind's indexing behavior by adding a `pagefind` configuration in your `site.json`. This allows you to control which content is indexed and how search works.
+
+#### Excluding Content from Search Index
+
+You can use the `exclude_selectors` option to exclude specific elements from the search index. This is useful if you are migrating from Algolia and want to reuse your existing CSS class selectors.
+
+In your `site.json`:
+
+```json
+{
+  "pagefind": {
+    "exclude_selectors": [".algolia-no-index", "[class*='algolia-no-index']"]
+  }
+}
+```
+
+This tells Pagefind to exclude any element with the `algolia-no-index` class (or containing it in a space-separated list) from the search index, similar to using `data-pagefind-ignore`.
+
+#### Limiting Which Pages Are Searchable
+
+You can use the `glob` option to limit which pages are indexed by Pagefind. This is useful when you want search results to only show pages from specific sections of your site.
+
+In your `site.json`:
+
+```json
+{
+  "pagefind": {
+    "glob": [
+      "devGuide",
+      "userGuide/*"
+    ]
+  }
+}
+```
+
+MarkBind supports glob patterns and will automatically append `.html` to your patterns if not specified. For example:
+- `"devGuide"` becomes `"devGuide/**/*.html"`
+- `"devGuide/*"` becomes `"devGuide/*.html"`
+- `"**/devGuide/**"` becomes `"**/devGuide/**/*.html"`
+- `"*.html"` remains `"*.html"` (no change needed)
+
+Only pages matching these glob patterns will appear in search results. This can be particularly useful for:
+- Multi-site setups where you want to search only specific sections
+- Including only certain directories from search results
+
+For more details on glob patterns, see the [Pagefind documentation](https://pagefind.app/docs/config-options/#glob).
+
+<panel header="Potential Future Enhancements">
+
+Additional Pagefind configuration options may be supported in future releases:
+
+- **`root_selector`**: Allows specifying a custom root element for indexing (default: `html`). Useful for sites with specific content containers.
+- **`force_language`**: Forces a specific language for indexing (e.g., `"en"`, `"pt"`). Improves search accuracy for multilingual sites.
+
+</panel>
+
+
+
+<br>
+
 ## Using External Search Services
 
 MarkBind sites can use Algolia Doc Search services easily via the Algolia plugin. Unlike the built-in search, Algolia provides full-text search. See the panel below for more info.

package-lock.json

@@ -50,12 +50,14 @@ expectedErrors.forEach((error, index) => {
   logger.info(`${index + 1}: ${error}`);
 });
 
+const GENERATED_DIRECTORIES_TO_IGNORE = ['**/pagefind/**'];
+
 testSites.forEach((siteName) => {
   console.log(`Running ${siteName} tests`);
   try {
     execSync(`node ${CLI_PATH} build ${siteName}`, execOptions);
     const siteIgnoredFiles = plantumlGeneratedFilesForTestSites[siteName];
-    compare(siteName, 'expected', '_site', siteIgnoredFiles);
+    compare(siteName, 'expected', '_site', siteIgnoredFiles, GENERATED_DIRECTORIES_TO_IGNORE);
   } catch (err) {
     if (_.isError(err)) {
       printFailedMessage(err, siteName);
@@ -74,7 +76,8 @@ testConvertSites.forEach((sitePath) => {
     execSync(`node ${CLI_PATH} init ${nonMarkBindSitePath} -c`, execOptions);
     execSync(`node ${CLI_PATH} build ${nonMarkBindSitePath}`, execOptions);
     const siteIgnoredFiles = plantumlGeneratedFilesForConvertSites[siteName];
-    compare(sitePath, 'expected', 'non_markbind_site/_site', siteIgnoredFiles);
+    compare(sitePath, 'expected', 'non_markbind_site/_site', siteIgnoredFiles,
+            GENERATED_DIRECTORIES_TO_IGNORE);
   } catch (err) {
     if (_.isError(err)) {
       printFailedMessage(err, sitePath);
@@ -98,7 +101,7 @@ testTemplateSites.forEach((templateAndSitePath) => {
     execSync(`node ${CLI_PATH} init ${siteCreationTempPath} --template ${flag}`, execOptions);
     execSync(`node ${CLI_PATH} build ${siteCreationTempPath}`, execOptions);
     const siteIgnoredFiles = plantumlGeneratedFilesForTemplateSites[siteName];
-    compare(sitePath, 'expected', 'tmp/_site', siteIgnoredFiles);
+    compare(sitePath, 'expected', 'tmp/_site', siteIgnoredFiles, GENERATED_DIRECTORIES_TO_IGNORE);
   } catch (err) {
     if (_.isError(err)) {
       printFailedMessage(err, sitePath);
@@ -141,7 +144,7 @@ function testEmptyDirectoryBuild() {
     } catch (err) {
       // Verify that test_empty directory remains empty using compare()
       try {
-        compare(siteRootName, 'expected', 'empty_dir', [], true);
+        compare(siteRootName, 'expected', 'empty_dir', [], [], true);
       } catch (compareErr) {
         if (_.isError(compareErr)) {
           printFailedMessage(compareErr, siteRootName);

packages/cli/test/functional/test.ts

@@ -13,6 +13,11 @@ const TEST_BLACKLIST = ignore().add([
   '*.log',
   '*.woff',
   '*.woff2',
+  '*.pf_fragment',
+  '*.pf_index',
+  '*.pf_meta',
+  '*.wasm.pagefind',
+  'wasm.unknown.pagefind',
 ]);
 
 const CRLF_REGEX = /\r\n/g;
@@ -54,10 +59,12 @@ function getDirectoryStructure(dirPath: string) {
  * @param {string} expectedSiteRelativePath - Relative path to expected site output (default: "expected")
  * @param {string} siteRelativePath - Relative path to actual generated site output (default: "_site")
  * @param {string[]} ignoredPaths - Specify any paths to ignore for comparison, but still check for existence.
+ * @param {string[]} ignoredDirectories - Specify any directories to ignore for comparison (e.g. 'pagefind')
  * @param {boolean} compareDirectories - Whether to compare directory structures (default: false)
  */
 function compare(root: string, expectedSiteRelativePath = 'expected', siteRelativePath = '_site',
-                 ignoredPaths: string[] = [], compareDirectories = false) {
+                 ignoredPaths: string[] = [], ignoredDirectories: string[] = [],
+                 compareDirectories = false) {
   const expectedDirectory = path.join(root, expectedSiteRelativePath);
   const actualDirectory = path.join(root, siteRelativePath);
 
@@ -73,8 +80,9 @@ function compare(root: string, expectedSiteRelativePath = 'expected', siteRelati
     }
   }
 
-  let expectedPaths = walkSync(expectedDirectory, { directories: false });
-  let actualPaths = walkSync(actualDirectory, { directories: false });
+  const walkSyncOptions = { directories: false, ignore: ignoredDirectories };
+  let expectedPaths = walkSync(expectedDirectory, walkSyncOptions);
+  let actualPaths = walkSync(actualDirectory, walkSyncOptions);
 
   // Vue render JS files (*.page-vue-render.js) are not committed to version control,
   // so we exclude them from the comparison to avoid false positive diffs.

packages/cli/test/functional/testUtil/compare.ts

@@ -16,6 +16,7 @@
   <link rel="stylesheet" href="/test_site/markbind/glyphicons/css/bootstrap-glyphicons.min.css">
   <link rel="stylesheet" href="/test_site/markbind/css/codeblock-dark.min.css">
   <link rel="stylesheet" href="/test_site/markbind/css/markbind.min.css">
+  <link rel="stylesheet" href="/test_site/markbind/pagefind/pagefind-ui.css">
   <link rel="stylesheet" href="/test_site/plugins/testMarkbindPlugin/testMarkbindPluginStylesheet.css">
   <link rel="stylesheet" href="/test_site/plugins/web3Form/web-3-form.css">
   <link rel="stylesheet" href="/test_site/plugins/markbind-plugin-anchors/markbind-plugin-anchors.css">
@@ -359,5 +360,6 @@ <h1 id="heading-in-footer-should-not-be-indexed">Heading in footer should not be
     });
 
   </script>
+<script src="/test_site/markbind/pagefind/pagefind-ui.js"></script>
 
 </html>

packages/cli/test/functional/test_site/expected/bugs/index.html

@@ -16,6 +16,7 @@
   <link rel="stylesheet" href="/test_site/markbind/glyphicons/css/bootstrap-glyphicons.min.css">
   <link rel="stylesheet" href="/test_site/markbind/css/codeblock-dark.min.css">
   <link rel="stylesheet" href="/test_site/markbind/css/markbind.min.css">
+  <link rel="stylesheet" href="/test_site/markbind/pagefind/pagefind-ui.css">
   <link rel="stylesheet" href="/test_site/plugins/testMarkbindPlugin/testMarkbindPluginStylesheet.css">
   <link rel="stylesheet" href="/test_site/plugins/web3Form/web-3-form.css">
   <link rel="stylesheet" href="/test_site/plugins/markbind-plugin-anchors/markbind-plugin-anchors.css">
@@ -1025,5 +1026,6 @@ <h1 id="heading-in-footer-should-not-be-indexed">Heading in footer should not be
     });
 
   </script>
+<script src="/test_site/markbind/pagefind/pagefind-ui.js"></script>
 
 </html>