Feature request: consolidate search index w/ markdown

[dreamorosi]

Use case

As discussed in #22, now that we've moved to consuming the Markdown versions of pages directly, there's an opportunity to consolidate the search index as well.

Currently our docs build looks like this:

When using the search_docs tool we use that lunr index and get back results that point to the HTML pages of the docs. That url is then converted to its Markdown version (by adding /index.md) and used by the fetch_doc_page tool to fetch the content.

We have an opportunity to add a new step to our docs build to create a dedicated index for these markdown files:

This would allow for greater consistency between search results and pages, as well as - in the future - give us full control over how content is indexed.

Solution/User Experience

The change would be completely internal, so customers should not be affected in any way.

Alternative solutions

Future readers

Please react with 👍 and your use case to help us understand customer demand.

[walmsles]

One thing to note is that MkDocs has a custom search index format they use since they have larger previews to add to the UX of web search capability. This means a vanilla lunr.js index is not possible from mkdocs. I support the decision to look at generating a seperate search index for markdown versions of docs and have been thinking that would be a good way to go.

Splitting the md processing for llm in this way also opens up wider use of other technologies to better support LLM outcomes.

[dreamorosi]

I've been thinking about this for a bit and using the MCP in some repositories, and came to the conclusion that the current search results are not helping LLM Agents perform well when dealing with details of the library. This is likely because despite using the search to identify a page section, the tool that fetches the page will always return the entire page - hurting both context pollution (#53) and information recall for the LLM.

Because of this, we should improve the fetch doc page tool to be able to return only the relevant portion(s) of documentation to any given query.

An naive way to do this would be to continue fetching the entire page, parse it locally, and return only the section but this has two problems: 1/ the fetch doc page tool doesn't have knowledge of the previous search since its only input is the doc page url, and 2/ that would be a waste of network bandwidth and time.

Since we control the documentation and its indexing, we could modify the search index to include the byte range of the content in each result. This way, provided that the fetch doc page tool is also modified to accept bytes_start and bytes_end optional parameters, we could fetch only the correct part of the doc.

This relies on the documentation CDN/Server supporting the HTTP Range header which both our Powertools for AWS and the wider AWS Documentation support. You can check by running this example curl command, which should return only a few lines of file:

curl -r 500-1000 https://docs.powertools.aws.dev/lambda/typescript/latest/features/logger/index.md > section.md

In order for this to work, we'll have to do a custom build of the search index (which is why I'm writing this under this issue). The new index should index items like this:

// sections is an array of processed doc sections

function buildSearchIndex(sections) {
  const idx = lunr(function() {
    this.ref('id');
    this.field('title', { boost: 10 });
    this.field('content');
    
    sections.forEach(section => {
      this.add({
        id: section.id,
        title: section.title,
        byteRange: `${section.startByte}-${section.endByte}`,
        filePath: section.filePath,
      });
    });
  });
  
  return idx;
}

This part of the work should happen in the aws-powertools/actions repository, since this is a shared logic that should work for all other Powertools for AWS repos, the rest of the indexing process should look something like:

• Find all markdown files in built version of the docs (possibly allow for an exclusion list via config file)
• Split each document into logical sections (by headings)
• Calculate and store byte ranges for each section together with id, title, etc.

Once that's built and deployed to our gamma stage of the docs, we can move onto working here and updating both tools of the MCP to support the new parameters. Putting on hold until then.