[FEATURE] Create automated documentation generation system for /docs/guides folder

[najeed]

Is your feature request related to a problem? Please describe.
With 4500+ scenarios across 44 industries and growing codebase, manually maintaining documentation in /docs/guides is becoming unsustainable. Documentation often becomes outdated, inconsistent, or incomplete as source code evolves.

Describe the solution you'd like
An automated documentation generation system that:

1. Pulls from Multiple Sources

   • Extracts docstrings from Python modules
   • Parses scenario JSON files for metadata
   • Aggregates industry/scenario statistics
   • Pulls configuration examples from source

2. Generates Comprehensive Guides

   • API reference documentation
   • Industry-specific guides with scenario counts
   • Integration tutorials with code examples
   • Configuration reference guides

3. Maintains Documentation Structure

   • Follows existing /docs/guides organization
   • Creates scalable folder structure for growth
   • Generates cross-references and navigation
   • Maintains consistent formatting and style

Proposed Implementation

# Automated generation commands
python scripts/generate_docs.py --all
python scripts/generate_docs.py --api-reference
python scripts/generate_docs.py --industry-guides
python scripts/generate_docs.py --scenarios-summary

# CI/CD integration
# Automatically update docs on code changes

Expected Output Structure

/docs/guides/
├── api/
│   ├── eval-runner.md           # Auto-generated from docstrings
│   ├── agents.md                # Agent interface documentation
│   └── metrics.md               # Metrics system reference
├── industries/
│   ├── healthcare.md            # Auto-generated scenario summaries
│   ├── manufacturing.md         # Industry-specific guides
│   └── finance.md               # With scenario counts and examples
├── integrations/
│   ├── langchain.md             # Integration examples from source
│   ├── openai.md                # Auto-generated integration guides
│   └── custom-agents.md         # Template generation
└── configuration/
    ├── evaluation-config.md     # Configuration reference
    ├── scenario-format.md       # JSON schema documentation
    └── deployment-guide.md      # Environment setup guides

Technical Requirements

• Parse Python docstrings using ast or inspect modules
• JSON schema extraction for scenario documentation
• Template-based markdown generation (Jinja2 or similar)
• Integration with existing /docs/guides structure
• Incremental updates (only regenerate changed sections)

Acceptance Criteria

• Scripts generate comprehensive API documentation from source
• Industry guides auto-update with scenario counts and examples
• Integration tutorials pull actual code examples
• Generated docs follow consistent formatting
• CI/CD integration updates docs on code changes
• Documentation cross-references work correctly
• Generated content integrates with existing manual docs

Benefits

• Always up-to-date documentation
• Consistent formatting and structure
• Reduced manual maintenance burden
• Scalable as project grows
• Better discoverability of features

Files to Create

• scripts/generate_docs.py (main generation script)
• scripts/doc_generators/ (modular generators)
• templates/ (markdown templates)
• .github/workflows/docs.yml (CI/CD integration)
• docs/AUTOMATION.md (system documentation)

Implementation Phases
Phase 1: API reference generation from docstrings
Phase 2: Industry guides with scenario statistics
Phase 3: Integration tutorials from source examples
Phase 4: CI/CD integration and automation

Additional context
This addresses the scalability challenge of maintaining comprehensive documentation for a rapidly growing project with 4500+ scenarios. Similar to documentation generation used by major open-source projects like Django, FastAPI, and TensorFlow.

Estimated Effort

• Large (2+ weeks)

Dependencies

• Python AST parsing libraries
• Jinja2 or similar templating engine
• JSON schema validation tools
• Markdown processing libraries