Machine-Readable Interfaces for HoloViz Libraries: Enabling Accurate AI Code Generation

[Azaya89]

Project

HoloViz

Summary

AI coding assistants frequently generate incorrect HoloViz code, using outdated APIs, invalid parameters, and fabricated functions. One reason for these issues is that LLMs (large language models) struggle to parse heavily nested HTML documentation structures effectively. This project will create machine-readable companion files (llms.txt and llms-full.txt) that sit alongside existing documentation, providing AI systems with clean, authoritative references for each HoloViz library. We'll implement these machine-readable interfaces across Panel, HoloViews, hvPlot, Datashader, and Param, while measuring improvements in AI code generation accuracy. Beyond reducing friction for new and experienced users, the project will produce reusable tools and patterns that other scientific Python projects can adopt and learn from to improve their AI compatibility.

Submitter

Isaiah Akorita

Project lead

@droumis

Community benefit

The Problem: Current LLMs struggle to extract accurate information from HTML documentation, leading to an all too common generation of invented APIs that don't exist. They also mix syntax from different library versions, propose invalid parameter combinations, and provide incorrect import paths and integration patterns. This isn't necessarily a documentation quality issue but rather a machine-readability problem. HTML pages include navigation elements, embedded notebooks, references to images, and JavaScript that confuse AI parsers, making it difficult for them to extract the authoritative information developers need.

Our Solution: We'll create dedicated machine-readable files that AI systems can efficiently consume without modifying the existing human-facing documentation. The llms.txt files will provide curated indexes of key documentation pages, while llms-full.txt files will contain clean, complete text extractions of all documentation. We'll also explore structured schemas in JSON or YAML formats for API definitions. These companion files will be automatically generated during the documentation build process and deployed alongside existing docs, giving AI systems direct access to authoritative information about HoloViz APIs, parameters, and usage patterns.

Broader Impact: This project's benefits extend far beyond HoloViz. We'll develop a reusable pipeline that other Sphinx-based docs projects could learn from and potentially adopt, along with a benchmarking framework to measure AI accuracy with Python libraries. The tools and patterns we create will provide a template for the scientific Python ecosystem to improve AI compatibility. By establishing best practices for machine-readable documentation and releasing our tools as open-source, we enable other NumFOCUS projects to explore implementation with similar improvements. This systematic approach to AI compatibility represents an evolution in how open-source projects can support the growing number of developers who rely on AI assistance for learning and productivity.

Amount requested (USD)

9000

Execution plan

Duration: 9 months (360 hours)

Months 1-3 — Foundation: Baseline Assessment and Infrastructure

Objective: Establish comprehensive baselines and build extraction infrastructure

Activities:

• Audit documentation structure across all HoloViz libraries, identifying extraction challenges
• Create benchmark suite (200+ test cases) covering API usage, parameters, workflows, and integration
• Establish baseline metrics (e.g. hallucination rate) using HoloViz's Lumen AI, which allows LLM configuration and testing with/without embedded documents, enabling direct before/after comparisons across multiple models
• Explore a Docling-based extraction pipeline for parsing complex Sphinx/notebook content into clean text
• Design llms.txt template structure optimized for AI consumption (hierarchical sections, concise summaries, prioritized links)
• Develop reusable Sphinx extension with automated generation hooks
• Build automated testing framework and metrics dashboard
• Prototype extraction on small documentation samples

Deliverables:

• Comprehensive baseline accuracy report
• Functional extraction pipeline
• Sphinx extension prototype
• Automated benchmark suite with dashboard
• Technical architecture documentation

Budget Allocation: 120 hours / $3,000

Months 4-6 — Core Implementation: Panel, HoloViews, and hvPlot

Objective: Implement machine-readable files for primary libraries with validation

Activities:

• Author curated llms.txt files for Panel, HoloViews, and hvPlot
• Generate comprehensive llms-full.txt using extraction pipeline
• Refine extraction based on library-specific needs, e.g.:
  • Panel: widget parameters and callback patterns
  • HoloViews: element types and options system
  • hvPlot: extensive kwargs and backend configurations
• Add CI job to rebuild and publish both files on every docs deploy.
• Iterate based on measured improvements
• Address extraction edge cases and optimize content structure

Deliverables:

• Machine-readable files for Panel, HoloViews, and hvPlot
• CI/CD pipeline with quality validation
• Performance improvement metrics
• Refined extraction pipeline based on real-world usage

Budget Allocation: 120 hours / $3,000

Months 7-9 — Expansion and Release

Objective: Expand ecosystem coverage, develop tools, and write documentation

Activities:

• Extend machine-readable file generation to Datashader, Param, and additional core HoloViz tools
• Resolve project-specific needs and run comprehensive validation
• Build and release public tools for other projects to learn from and potentially adopt
• Write documentation, guides, and tutorials
• Document maintenance guidance for contributors (how to update llms.txt, regenerate llms-full.txt, and interpret CI checks)
• Engage the community through announcement and outreach materials
• Final optimizations based on complete testing and community feedback (reorder sections, improve summaries, add missing “must-know” links).

Deliverables:

• Machine-readable files for all core HoloViz libraries
• Publicly released Sphinx docs hook, extension, or package
• Final accuracy report with quantified improvements
• Implementation guide and blog post to raise awareness
• Finalized grant report and presentation

Budget Allocation: 120 hours / $3,000

Success Metrics

We will measure improvement across multiple dimensions:

• Reduction in non-existent API suggestions
• Increase in valid parameter usage
• Improvement in executable code generation
• Better handling of version-specific features

Final Key Deliverables

• Working tool that emits llms.txt, and llms-full.txt during docs builds.
• Curated llms.txt files per repo with short, actionable page summaries.
• CI validation with, e.g., link, size, freshness checks
• Complete documentation for maintainers and new adopters (how it works, which parameters to adjust and why).

[dpshelio]

🎉 Congratulations, this project has been selected this time (by the random script).

[RitikaSen27]

Hi,

I’m very interested in contributing to this project. I’ve read through the full proposal and really like the idea of improving AI compatibility through machine-readable documentation rather than altering the human-facing docs. The framing around llms.txt / llms-full.txt as companion artifacts and the emphasis on measurable improvements resonates strongly with me.

I’m particularly interested in helping with:

Auditing Sphinx-based documentation structures and identifying extraction pain points

Working on or extending the extraction pipeline (e.g. Docling-based or similar) to produce clean, authoritative text

Designing and validating the llms.txt structure so it’s genuinely useful for LLMs (clear hierarchy, concise summaries, priority ordering)

Benchmarking and validation: writing or expanding test cases to measure hallucination rate, invalid API usage, and executable code accuracy

I understand this is a funded, multi-month effort, and I wanted to ask how best I can contribute — whether that’s by taking ownership of a specific library (e.g. Panel or hvPlot), helping build the extraction/CI tooling, or starting with a smaller scoped task aligned with the Phase 1 foundation work.

Please let me know if there’s a preferred entry point or any design constraints I should be aware of before starting. I’d be happy to follow the project lead’s direction.

Thanks!

[droumis]

Hi @RitikaSen27 , thanks for expressing your interest! As we make some initial progress in the next couple of weeks, we'll be in a better position to determine what role or tasks there might be for an additional contributor.

[RitikaSen27]

okay thank you, please let me know .