feat(docs): auto-generate database documentation from lib.py

[rusackas]

Summary

This PR introduces an automated database documentation system that generates documentation pages from engine spec metadata attributes. Each database engine spec now contains its own documentation metadata, providing a single source of truth.

Key Features

• Single source of truth: All database documentation lives in each engine spec's metadata attribute (removed 1150+ line DATABASE_DOCS dict from lib.py)
• Auto-generated pages: Individual MDX pages for each database with connection strings, drivers, auth methods
• Overview table: Searchable/filterable table showing all databases with scores, features, time grains
• Metadata linter: Script to track completeness and identify specs needing documentation
• PostgreSQL-compatible DBs: TimescaleDB, YugabyteDB, Hologres now have their own engine specs
• Simplified build: Generator script parses engine spec metadata via AST - no Flask context required for CI

Architecture

superset/db_engine_specs/
├── postgres.py          # PostgresEngineSpec with metadata = {...}
├── mysql.py             # MySQLEngineSpec with metadata = {...}
├── timescaledb.py       # NEW: PostgreSQL-compatible stub spec
├── yugabytedb.py        # NEW: PostgreSQL-compatible stub spec
├── hologres.py          # NEW: PostgreSQL-compatible stub spec
├── arc.py               # NEW: Stub spec for Arc
├── d1.py                # NEW: Stub spec for Cloudflare D1
├── lint_metadata.py     # Metadata completeness linter
├── METADATA_STATUS.md   # Auto-generated completeness report
└── README.md            # Updated with "How to Add a Database" guide

How to Add a New Database

1. Create a new file in superset/db_engine_specs/ (e.g., mydatabase.py)
2. Add a metadata attribute with required fields:

   from superset.db_engine_specs.base import BaseEngineSpec, DatabaseCategory
   
   class MyDatabaseEngineSpec(BaseEngineSpec):
       engine = "mydatabase"
       engine_name = "My Database"
   
       metadata = {
           "description": "Brief description of the database.",
           "category": DatabaseCategory.TRADITIONAL_RDBMS,
           "pypi_packages": ["my-driver"],
           "connection_string": "mydb://{username}:{password}@{host}:{port}/{database}",
           "logo": "mydb.svg",
           "homepage_url": "https://mydb.example.com/",
           "default_port": 5432,
       }

3. Run the linter: python superset/db_engine_specs/lint_metadata.py
4. Add a logo (optional) in docs/static/img/databases/

Changes

New Files:

• superset/db_engine_specs/arc.py - Arc data platform stub spec
• superset/db_engine_specs/d1.py - Cloudflare D1 stub spec
• superset/db_engine_specs/hologres.py - Alibaba Cloud Hologres (PostgreSQL-compatible)
• superset/db_engine_specs/timescaledb.py - TimescaleDB (PostgreSQL-compatible)
• superset/db_engine_specs/yugabytedb.py - YugabyteDB (PostgreSQL-compatible)
• superset/db_engine_specs/lint_metadata.py - Metadata completeness linter
• superset/db_engine_specs/METADATA_STATUS.md - Auto-generated status report

Modified Files:

• superset/db_engine_specs/lib.py - Removed DATABASE_DOCS dict (~1150 lines), updated get_documentation_metadata()
• superset/db_engine_specs/README.md - Added comprehensive "How to Add a Database" guide
• superset/db_engine_specs/*.py - Added metadata attributes to 60+ engine specs
• docs/scripts/generate-database-docs.mjs - Simplified to read from engine spec metadata via AST (removed DATABASE_DOCS fallback)

Documentation Build Modes

The generate-database-docs.mjs script supports two modes:

1. Full mode (with Flask context): Runs diagnose() to get detailed feature scores. Requires Superset installed locally.
2. AST fallback (CI/Netlify): Parses engine spec files directly to extract metadata attributes. Works without Flask.

Metadata Completeness

Current status (63 engine specs with metadata):

• All have required fields (description, category, pypi_packages, connection_string)
• Most have recommended fields (logo, homepage_url, default_port)

Run python superset/db_engine_specs/lint_metadata.py to see the full report.

Screenshots

Test Plan

1. Run python superset/db_engine_specs/lint_metadata.py to verify metadata extraction
2. Run cd docs && yarn build to verify documentation generation
3. Check the generated databases.json for all 63 databases
4. Verify each database page renders correctly at /docs/databases/<database-name>

[codeant-ai-for-open-source]

CodeAnt AI is running Incremental review

---

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

[codeant-ai-for-open-source]

CodeAnt AI is running Incremental review

---

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn

[netlify]

✅ Deploy Preview for superset-docs-preview ready!

Name	Link
🔨 Latest commit	5873ead
🔍 Latest deploy log	https://app.netlify.com/projects/superset-docs-preview/deploys/696eb5a0981f24000849ed27
😎 Deploy Preview	https://deploy-preview-36805--superset-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[codeant-ai-for-open-source]

CodeAnt AI is running Incremental review

---

Thanks for using CodeAnt! 🎉

We're free for open-source projects. if you're enjoying it, help us grow by sharing.

Share on X ·
Reddit ·
LinkedIn