Header DocSearch Module

[giscus[bot]]

Header DocSearch Module

DocSearch is a free online search service for open source documentation sites.

https://hbstack.dev/modules/header/docsearch/

[FuadEfendi]

Hi Razon,

I want to implement custom Algolia Search but I cannot understand whole picture, how to integrate it to current "header"; how to make "Search Ctrl K" in the header to open popup window. Could you please provide path I can follow to implement this widget as "hbstack" module?
I already have some JavaScript working code but cannot understand how to integrate it per HBStack standards

Thanks

[FuadEfendi]

The only issue is that it Widget shows 1 search result only with few sections (subtitles) of the same article. For example, search for "budget" directly in Algolia Console has 476 hits (23,000 records). But this is weakness of their DocSearch maybe, I am not sure, still need research...
After more play: "price budget" shows scrollbar with about 25 search results. But search for "budget" shows 1 main and 4 subsections, which corresponds to https://docsearch.algolia.com/docs/api#maxresultspergroup
So, it is "DocSearch" JS client from Algolia designed such way.

[razonyang]

😕 that would be simple if this module can directly connecting with Algolia projects. So the problem may be caused by the index file, as I know, different heading hierarchy should have unique index record.

For example, the following page

---
title: FOO
---

## FIZZ

## BAR

should produce 3 indices,

1. Only lvl1 (FOO)
2. lvl1 + lvl2 (FOO + FIZZ)
3. lvl1 + lvl2 (FOO + BAR)

But those are all about guessing, you might need to asking their support about how to generate indices.
Or building search ui

[FuadEfendi]

But yes, I connected this module to my custom index: 22,000 records in Algolia, based upon approx. 1,500 "index.md" files (so that each heading level has separate index entry; few entries per article). Initially it didn't work but then I realized I had to go to Algolia Console and configure explicitly searchable attributes.
By default, DocSearch is Crawler based and free for 10,000 entries; I used it few years ago for 20,000 entries: Netlify plugin has default 20,000 entries. But few of my sites have more that 50k pages, it would take from few days to few weeks to crawl.
So, front-end DocSearch works just fine with custom index (without crawler); I only had to research index format.

[FuadEfendi]

For this:

Only lvl1 (FOO)
lvl1 + lvl2 (FOO + FIZZ)
lvl1 + lvl2 (FOO + BAR)

There is "content" field too, and it should work, "FOO BAR" will be found in content field. Record size is limited to 10,000 in Algolia, so I'll have to truncate content field. And also, I didn't do research in depth yet, but ranking, searchable fields, etc. could be tuned in Algolia.

Just basic first step, not ideal, but I made it work...

Sooner or later I want to create custom widget, similar to Fuse.js based search, but working with Algolia, I have some working prototype but it is non-standard for hbstack

[FuadEfendi]

I caught which fields DocSearch queries via Chrome debug, and then created custom index withe these fields; only after that I found it in formal Algolia docs

URL:

https://gh97vgo5v0-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20JavaScript%20(4.19.1)%3B%20Browser%20(lite)%3B%20docsearch%20(3.6.0)%3B%20docsearch-react%20(3.6.0)%3B%20docsearch.js%20(3.6.0)&x-algolia-api-key=16d3d3977efd76636d7d2e0145ebc6ad&x-algolia-application-id=GH97VGO5V0

Payload:

{"requests":[{"query":"price","indexName":"cpa_exams_mastery-v1","params":"attributesToRetrieve: ["hierarchy.lvl0","hierarchy.lvl1","hierarchy.lvl2","hierarchy.lvl3","hierarchy.lvl4","hierarchy.lvl5","hierarchy.lvl6","content","type","url"]
attributesToSnippet: ["hierarchy.lvl1:5","hierarchy.lvl2:5","hierarchy.lvl3:5","hierarchy.lvl4:5","hierarchy.lvl5:5","hierarchy.lvl6:5","content:5"]
snippetEllipsisText: …
highlightPreTag: <mark>
highlightPostTag: </mark>
hitsPerPage: 20
clickAnalytics: false"}]}

[FuadEfendi]

Some screenshots,

Last one, "Budget price", has scrollbar; it shows indexed KaTeXX formula snippet too.

[FuadEfendi]

I implemented it for UltimateLexicon.com - English & Beyond, just a dictionary (I am going to make it open source project soon)

It took 24 minutes to build locally (over 200k pages), and few hours to upload to CDN. Algolia index size: 3,500,000 approximately. I wrote small Java application which uploads index in small batches; I could not upload large 2.3Gb JSON file. I believe I messed up with index; I have experience with SOLR, I do not have to create separate entry per document subsection; I think I'll tune it more later, maybe separate entry per Heading Level 1,...6 and "content" will be just snippet under heading (currently it is whole doc) then probably it would work much better.

It works!