Merge pull request #3989 from ruby/claude/pagefind-inline-search-i18n-49fa5f

hsbt · web-flow · commit b5cff3d3a0f5 · 2026-07-28T11:59:18.000+09:00
Add Pagefind search to the site
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -11,10 +11,10 @@ This is the Jekyll-based source for the official Ruby programming language websi
 ### Jekyll Site Operations
 
 ```bash
-# Build the site (takes several minutes)
+# Build the site and its search index (takes several minutes)
 bundle exec rake build
 
-# Serve locally at http://localhost:4000/
+# Build, watch and serve at http://localhost:4000/, search index included
 bundle exec rake serve
 
 # Alternative: Jekyll direct serve with incremental builds
@@ -40,6 +40,7 @@ bundle exec rake test
 # Run individual test suites
 bundle exec rake test-news-plugin     # News archive plugin tests
 bundle exec rake test-linter          # Linter library tests
+bundle exec rake test-search-index    # Search index library tests
 
 # Linting
 bundle exec rake lint                 # Markdown linter
@@ -109,6 +110,31 @@ The news system is powered by a custom Jekyll plugin (`_plugins/news.rb`):
 - **Archive pages**: Index, yearly archives, monthly archives
 - **RSS feeds**: Generated per language via `news_feed.rss` layout
 
+### Search (Pagefind)
+
+Search is client-side and needs no backend. `_plugins/search_index.rb` hooks
+Jekyll's `post_write` and calls `lib/search_index.rb`, so every build and every
+regeneration under `rake serve` ends with a matching index. It has to run after
+the write because Pagefind reads the generated HTML, and it has to run every
+time because Jekyll deletes destination files with no source counterpart, the
+previous bundle included. Without Node installed the hook warns and skips.
+
+- **Indexed region**: `_includes/search_body.html` emits `data-pagefind-body` on
+  the `<article>` of `page.html` and `news_post.html`. Everything outside it, and
+  anything marked `data-pagefind-ignore` inside it, stays out of results.
+- **Per-language indexes**: Pagefind splits the index by the `lang` attribute on
+  `<html>` and the browser loads the one matching the page, so a translation only
+  ever finds its own pages. The same attribute picks the interface language, so
+  no UI strings live in `_data/locales`.
+- **Unsupported languages**: `search.unsupported` in `_config.yml` lists the
+  languages Pagefind cannot index well (currently `bg`). Their pages get no
+  `data-pagefind-body`, and `lib/search_index.rb` rewrites `pagefind-entry.json`
+  so they resolve to `search.fallback`. Without that rewrite Pagefind would fall
+  back to whichever index has the most pages, which is not a language we choose.
+- **UI**: `_includes/search.html` (button, modal, config) and
+  `_includes/search_assets.html` (bundle, loaded before `compiled.css` so the
+  site's `--pf-*` overrides win).
+
 ### Linter (`lib/linter.rb`)
 
 Enforces strict content quality rules:
@@ -178,6 +204,7 @@ lang: en
 **Node (package.json):**
 - `tailwindcss` - CSS framework
 - `@tailwindcss/typography` - Prose styling plugin
+- `pagefind` - Static search index and UI
 
 ## Important Conventions
 
diff --git a/README.md b/README.md
@@ -138,6 +138,28 @@ npm run watch-css   # watch and rebuild CSS automatically
 **Note:** You need to have Node.js installed to run these commands.
 
 
+## Search
+
+Search is [Pagefind](https://pagefind.app/), which indexes the HTML Jekyll
+produced rather than the Markdown behind it, so it needs no server of its own.
+`_plugins/search_index.rb` runs it whenever Jekyll finishes writing the site,
+which means `rake build` and `rake serve` both leave you with a search index
+matching the site they just produced. Nothing extra to run.
+
+Indexing adds a few seconds to a build and needs Node, so run `npm install`
+first. Without it `bundle exec jekyll build` still finishes and only reports that
+the index was skipped, which is enough for editing content. The `rake` tasks
+stop earlier than that, since they build the CSS with Tailwind first.
+
+Pagefind builds one index per language and picks one by the `lang` attribute of
+the page the visitor is on, so each translation searches its own pages and reads
+its own interface labels. Bulgarian is the exception: Pagefind has neither word
+stemming nor interface translations for it, so `search.unsupported` in
+`_config.yml` keeps its pages out of the index and its search runs against
+English instead. Adding a language there is all it takes to do the same for
+another one.
+
+
 ## Testing
 
 Besides generating and previewing the site
diff --git a/Rakefile b/Rakefile
@@ -18,8 +18,8 @@ task :"build-css" do
   sh "npm run build-css"
 end
 
-desc "Run tests (test-linter, lint, build)"
-task test: %i[test-news-plugin test-html-lang-plugin test-linter lint build]
+desc "Run every test suite, the markdown linter and a full build"
+task test: %i[test-news-plugin test-html-lang-plugin test-linter test-search-index lint build]
 
 desc "Build the Jekyll site"
 task build: :"build-css" do
@@ -32,7 +32,13 @@ desc "Serve the Jekyll site locally"
 task serve: :"build-css" do
   require "jekyll"
 
-  Jekyll::Commands::Serve.process({})
+  # Same pair `jekyll serve` runs. Serving on its own only hands out whatever
+  # _site already holds, which leaves a fresh clone with nothing to serve and
+  # never picks up an edit.
+  options = { "serving" => true, "watch" => true }
+
+  Jekyll::Commands::Build.process(options)
+  Jekyll::Commands::Serve.process(options)
 end
 
 namespace :new_post do
@@ -142,3 +148,11 @@ Rake::TestTask.new(:"test-html-lang-plugin") do |t|
   t.test_files = FileList['test/test_plugin_html_lang.rb']
   t.verbose = true
 end
+
+require "rake/testtask"
+Rake::TestTask.new(:"test-search-index") do |t|
+  t.description = "Run tests for the search index library"
+  t.libs = ["test", "lib"]
+  t.test_files = FileList['test/test_search_index.rb']
+  t.verbose = true
+end
diff --git a/_config.yml b/_config.yml
@@ -20,6 +20,16 @@ exclude:
   - vendor
   - CLAUDE.md
 
+# Pagefind builds one search index per language and the browser picks one by the
+# `lang` of the page it is on. Pagefind has neither word stemming nor interface
+# translations for the languages under `unsupported`, so their pages are left
+# out of the index and their search runs against `fallback` instead. Both the
+# layouts and lib/search_index.rb read these keys.
+search:
+  fallback: en
+  unsupported:
+    - bg
+
 url: https://www.ruby-lang.org
 
 license:
diff --git a/_data/locales/bg.yml b/_data/locales/bg.yml
@@ -166,6 +166,11 @@ feed:
   description: Последните новини от ruby-lang.org.
   lang_code: bg
 
+search:
+  fallback_notice: >-
+    Търсенето не поддържа български, затова обхваща страниците на английски
+    език. Въведете заявката си на английски.
+
 news:
   other_news: Други новини
   more_news: Още новини...
diff --git a/_data/locales/en.yml b/_data/locales/en.yml
@@ -188,6 +188,13 @@ feed:
   description: The latest news from ruby-lang.org.
   lang_code: en-US
 
+# Shown inside the search modal on pages whose language is listed under
+# `search.unsupported` in _config.yml. Other languages never see it.
+search:
+  fallback_notice: >-
+    Search is not available in this language, so it looks through the English
+    pages. Type your search in English.
+
 news:
   other_news: Other News
   more_news: More News...
diff --git a/_includes/header.html b/_includes/header.html
@@ -26,8 +26,11 @@
         {% endfor %}
       </nav>
 
-      <!-- Right section: Language selector + Theme toggle + Mobile menu button -->
+      <!-- Right section: Search + Language selector + Theme toggle + Mobile menu button -->
       <div class="flex items-center gap-0.5 sm:gap-2">
+        <!-- Search -->
+        {% include search.html %}
+
         <!-- Language Selector -->
         {% include language_selector.html %}
 
@@ -51,6 +54,11 @@
   <!-- Mobile Navigation Menu -->
   <div id="mobile-menu" class="hidden lg:hidden border-t border-stone-200 dark:border-stone-700">
     <nav class="container mx-auto px-4 py-4">
+      <!-- Search, for the widths where the top bar has no room for it -->
+      <div class="search-trigger-menu mb-3">
+        <pagefind-modal-trigger></pagefind-modal-trigger>
+      </div>
+
       {% for item in nav_items %}
       <a href="{{ item.url }}"
          class="block px-5 py-2 text-sm font-bold text-stone-900 dark:text-stone-300 hover:text-semantic-text-hovered dark:hover:text-ruby-400 hover:bg-stone-100 dark:hover:bg-stone-800 rounded-md transition-colors mb-2">
diff --git a/_includes/pagination.html b/_includes/pagination.html
@@ -8,7 +8,7 @@
   {% assign header_parts = page.header | split: '</div>' %}
   {% for part in header_parts %}
     {% if part contains 'class="multi-page"' %}
-      <div class="pagination-footer">
+      <div class="pagination-footer" data-pagefind-ignore>
         {{ part }}</div>
       </div>
       {% break %}
diff --git a/_includes/recent_news.html b/_includes/recent_news.html
@@ -26,7 +26,7 @@
 {% endfor %}
 
 {% if recent_posts.size > 0 %}
-<section class="not-prose mt-16 p-6 md:p-8 border border-stone-200 dark:border-stone-700 rounded-xl">
+<section class="not-prose mt-16 p-6 md:p-8 border border-stone-200 dark:border-stone-700 rounded-xl" data-pagefind-ignore>
   <h2 class="text-2xl font-bold mb-6">
     <a href="/{{ page.lang }}/news/" class="text-stone-900 dark:text-stone-100 hover:text-ruby-600 dark:hover:text-ruby-400 transition-colors inline-flex items-center">
       {{ news_locale.recent_news }}<span class="icon-chevron-right text-2xl" aria-hidden="true"></span>
diff --git a/_includes/search.html b/_includes/search.html
@@ -0,0 +1,49 @@
+{%- comment -%}
+Site search. The button opens a modal; Cmd+K or Ctrl+K opens it from anywhere.
+
+Pagefind picks both the index it searches and the language of its own interface
+from the `lang` attribute on `<html>`, so every translation searches its own
+pages and reads its own labels without anything to maintain here. The `lang`
+override below is for the languages under `search.unsupported` in _config.yml,
+whose pages are not indexed: lib/search_index.rb points their index at
+`search.fallback` and this points their interface at the same language, so the
+two cannot drift apart.
+
+`pagefind-config` has to come first. Its attributes are read once, when the
+first component of the instance connects. The trigger here is the one in the top
+bar; below 640px it gives way to the one in the mobile menu, which shares this
+modal through the default instance.
+
+An empty `pagefind-modal` builds its own contents. Those languages that fall
+back get the same parts written out so a notice can sit above the results,
+because otherwise a search in their own language just returns nothing and reads
+as a broken feature.
+{%- endcomment -%}
+{%- assign search_lang = page.lang -%}
+{%- assign falls_back = false -%}
+{%- if site.search.unsupported contains page.lang -%}
+  {%- assign search_lang = site.search.fallback -%}
+  {%- assign falls_back = true -%}
+{%- endif -%}
+<pagefind-config highlight-param="highlight" lang="{{ search_lang | to_html_lang }}"></pagefind-config>
+<div class="search-trigger-bar">
+  <pagefind-modal-trigger compact></pagefind-modal-trigger>
+</div>
+{%- if falls_back -%}
+  {%- assign search_locale = site.data.locales[page.lang].search | default: site.data.locales.en.search -%}
+<pagefind-modal reset-on-close>
+  <pagefind-modal-header>
+    <pagefind-input></pagefind-input>
+  </pagefind-modal-header>
+  <pagefind-modal-body>
+    <p class="search-notice">{{ search_locale.fallback_notice }}</p>
+    <pagefind-summary></pagefind-summary>
+    <pagefind-results></pagefind-results>
+  </pagefind-modal-body>
+  <pagefind-modal-footer>
+    <pagefind-keyboard-hints></pagefind-keyboard-hints>
+  </pagefind-modal-footer>
+</pagefind-modal>
+{%- else -%}
+<pagefind-modal reset-on-close></pagefind-modal>
+{%- endif -%}
diff --git a/_includes/search_assets.html b/_includes/search_assets.html
@@ -0,0 +1,23 @@
+{%- comment -%}
+Pagefind's search bundle, written to /pagefind/ by _plugins/search_index.rb once
+Jekyll has finished writing the site. Every build produces it, `rake serve` and
+its regenerations included, so these files are missing only when the index was
+skipped for want of Node.
+
+This has to come before compiled.css: Pagefind ships its `--pf-*` defaults on
+`:root`, the same specificity stylesheets/components/search.css uses to override
+them, so whichever stylesheet loads last would win.
+{%- endcomment -%}
+<link rel="stylesheet" type="text/css" href="/pagefind/pagefind-component-ui.css">
+<script type="module" src="/pagefind/pagefind-component-ui.js"></script>
+<script type="module">
+  // pagefind-highlight.js carries mark.js with it, so it is only worth loading
+  // on the pages a search result linked to. A build that skipped the index has
+  // no such file, and highlighting is not worth an unhandled rejection.
+  if (new URLSearchParams(window.location.search).has('highlight')) {
+    try {
+      await import('/pagefind/pagefind-highlight.js');
+      new PagefindHighlight({ highlightParam: 'highlight' });
+    } catch {}
+  }
+</script>
diff --git a/_includes/search_body.html b/_includes/search_body.html
@@ -0,0 +1,17 @@
+{%- comment -%}
+Marks the element this is placed on as the only region Pagefind indexes, which
+keeps navigation, the table of contents and the footer out of search results.
+
+Pages in a language listed under `search.unsupported` in _config.yml emit
+nothing here. Once Pagefind has seen a `data-pagefind-body` anywhere on the site
+it skips every page without one, so leaving the attribute off is what keeps
+those pages out of the index. lib/search_index.rb then points the language at
+`search.fallback` so its visitors still get results.
+
+A page can opt out on its own with `searchable: false` in its front matter.
+{%- endcomment -%}
+{%- unless site.search.unsupported contains page.lang -%}
+  {%- if page.searchable != false -%}
+data-pagefind-body data-pagefind-meta="title:{{ page.title | strip_html | escape }}"
+  {%- endif -%}
+{%- endunless -%}
diff --git a/_layouts/default.html b/_layouts/default.html
@@ -62,6 +62,7 @@
   <meta itemprop="description" content="{{ meta_description }}">
   <meta itemprop="image" content="{{ site.url }}/images/og-image.png">
 
+  {% include search_assets.html %}
   <link rel="stylesheet" type="text/css" href="/stylesheets/compiled.css">
   <link rel="preconnect" href="https://fonts.googleapis.com">
   <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
diff --git a/_layouts/homepage.html b/_layouts/homepage.html
@@ -39,6 +39,7 @@
   <meta itemprop="description" content="{{ meta_description }}">
   <meta itemprop="image" content="{{ site.url }}/images/og-image.png">
 
+  {% include search_assets.html %}
   <link rel="stylesheet" type="text/css" href="/stylesheets/compiled.css">
   <link rel="preconnect" href="https://fonts.googleapis.com">
   <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
diff --git a/_layouts/news_post.html b/_layouts/news_post.html
@@ -45,7 +45,7 @@
       </div>
 
       <!-- Main content -->
-      <article id="content-wrapper" class="prose md:px-4 lg:px-6">
+      <article id="content-wrapper" class="prose md:px-4 lg:px-6" {% include search_body.html %}>
         {% include title.html %}
 
         <div id="content">
diff --git a/_layouts/page.html b/_layouts/page.html
@@ -34,7 +34,7 @@
   {% if is_homepage %}
     <!-- Home page: Single column layout -->
     <div class="max-w-4xl mx-auto">
-      <article id="content-wrapper" class="prose">
+      <article id="content-wrapper" class="prose" {% include search_body.html %}>
         {% include title.html %}
 
         <div id="content">
@@ -61,7 +61,7 @@
       </div>
 
       <!-- Main content -->
-      <article id="content-wrapper" class="prose md:px-4 lg:px-6">
+      <article id="content-wrapper" class="prose md:px-4 lg:px-6" {% include search_body.html %}>
         {% include title.html %}
 
         <div id="content">
diff --git a/_plugins/search_index.rb b/_plugins/search_index.rb
@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+require_relative "../lib/search_index"
+
+# Pagefind indexes the HTML Jekyll wrote, so it hangs off post_write rather than
+# off a Rake task. That way `rake build`, `rake serve` and a bare `jekyll build`
+# all end with a search index that matches the site they just produced.
+Jekyll::Hooks.register :site, :post_write do |site|
+  entry = SearchIndex.new(
+    bundle_dir: File.join(site.dest, "pagefind"),
+    search_config: site.config["search"],
+  ).build
+
+  next unless entry
+
+  # The languages that fall back share another language's index, so count the
+  # indexes rather than the keys pointing at them.
+  indexes = entry["languages"].values.uniq
+  pages = indexes.sum { |index| index["page_count"] }
+  Jekyll.logger.info "Search index:", "#{indexes.size} languages, #{pages} pages"
+end
diff --git a/admin/index.md b/admin/index.md
@@ -2,6 +2,7 @@
 layout: page
 title: "Admin Area"
 lang: en
+searchable: false
 ---
 
 - [News Post Translation Status](translation-status)
diff --git a/javascripts/theme-toggle.js b/javascripts/theme-toggle.js
@@ -35,6 +35,14 @@
         root.classList.remove('dark');
       }
     }
+
+    // Pagefind's search components read `data-pf-theme` and nothing else, so
+    // derive it from the class rather than repeating the decision above.
+    if (root.classList.contains('dark')) {
+      root.setAttribute('data-pf-theme', 'dark');
+    } else {
+      root.removeAttribute('data-pf-theme');
+    }
   }
 
   // Update toggle button icon
diff --git a/lib/search_index.rb b/lib/search_index.rb
diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
diff --git a/stylesheets/components/search.css b/stylesheets/components/search.css
diff --git a/stylesheets/tailwind.css b/stylesheets/tailwind.css
diff --git a/test/test_search_index.rb b/test/test_search_index.rb
