Easier way to shard documents

[ksylvan]

See issue #110

Here's the demonstration chat of talking to the agent and having it create a much easier-to-process sharding output:

https://g.co/gemini/share/ab3ff0e8f09d

And then I copy the whole output (using the "..." Copy that preserved the Markdown formatting of the entire output) and run my Python extraction script:

(fabric-mcp) kayvan@dharma src/fabric-mcp [develop?] $ python scripts/extract_files.py temp_output input.txt
Written: temp_output/docs/epic-1.md
Written: temp_output/docs/epic-2.md
Written: temp_output/docs/epic-3.md
Written: temp_output/docs/epic-4.md

And voila! All the epic files are there. Now I look them over, fix the heading levels, and pop them into my docs/ directory.

#!/usr/bin/env python3
"""
Script to extract files from a specially formatted text file.
The script processes a file that contains embedded file contents, extracts them,
and writes them to specified locations in a destination directory.
"""

import argparse
import os
import re
import sys


def main():
    """Main function to handle argument parsing and file processing."""
    # Parse command line arguments
    parser = argparse.ArgumentParser(
        description="Extract files from a formatted text file."
    )
    parser.add_argument(
        "dest_dir", help="Destination directory where extracted files will be written"
    )
    parser.add_argument(
        "input_file", help="Input file containing file definitions and contents"
    )

    args = parser.parse_args()

    # Make sure destination directory exists
    try:
        os.makedirs(args.dest_dir, exist_ok=True)
    except OSError as e:
        print(
            f"Error creating destination directory '{args.dest_dir}': {e}",
            file=sys.stderr,
        )
        return 1

    # Try to open and process the input file
    try:
        process_input_file(args.input_file, args.dest_dir)
    except (FileNotFoundError, OSError, UnicodeDecodeError) as e:
        print(f"Error processing input file: {e}", file=sys.stderr)
        return 1

    return 0


def process_input_file(input_file: str, dest_dir: str):
    """
    Process the input file and extract embedded files.

    Args:
        input_file (str): Path to the input file
        dest_dir (str): Destination directory for extracted files
    """
    file_pattern = re.compile(r"^\*\*filename: (.*)\*\*$")
    found_first_file = False
    current_filename = ""
    collecting_content = False
    current_content = []

    try:
        with open(input_file, encoding="utf-8") as f:
            for line in f:
                line = line.rstrip("\n")

                # Check if this line defines a filename
                match = file_pattern.match(line)
                if match:
                    # If we were already collecting content, write the previous file
                    if collecting_content and current_filename:
                        write_file(current_filename, current_content, dest_dir)

                    # Start collecting for a new file
                    found_first_file = True
                    current_filename = match.group(1)
                    collecting_content = False
                    current_content: list[str] = []
                    continue

                # Skip everything until we find the first filename
                if not found_first_file:
                    continue

                # Check for the start/end markers of file content
                if line.startswith("```"):
                    if not collecting_content:
                        # Start collecting content
                        collecting_content = True
                    else:
                        # End of content section, write the file
                        write_file(current_filename, current_content, dest_dir)
                        collecting_content = False
                        current_content = []
                        current_filename = ""
                    continue

                # If we're collecting content, add this line
                if collecting_content:
                    current_content.append(line)

        # Handle case where the file ends while still collecting content
        if collecting_content and current_filename:
            write_file(current_filename, current_content, dest_dir)

    except FileNotFoundError:
        print(f"Input file not found: {input_file}", file=sys.stderr)
        raise
    except OSError as e:
        print(f"Error reading input file: {e}", file=sys.stderr)
        raise


def write_file(filename: str, content: list[str], base_dir: str):
    """
    Write content to a file, creating any necessary directories.

    Args:
        filename (str): Path to the output file, relative to base_dir
        content (list): List of lines to write to the file
        base_dir (str): Base directory for the output file
    """
    # Replace @@@ with ``` in the content
    processed_content = [line.replace("@@@", "```") for line in content]

    # Construct the full path
    full_path = os.path.join(base_dir, filename)

    # Create parent directories if needed
    os.makedirs(os.path.dirname(full_path), exist_ok=True)

    # Write the file
    try:
        with open(full_path, "w", encoding="utf-8") as f:
            for line in processed_content:
                f.write(line + "\n")
        print(f"Written: {full_path}")
    except OSError as e:
        print(f"Error writing file {full_path}: {e}", file=sys.stderr)
        raise


if __name__ == "__main__":
    sys.exit(main())

The prompt I used is here as a text file:

sharding-prompt.txt

[ksylvan]

The better way to accomplish this is to embed comments in the original documents that include the target filenames, which a post-processing sharding script could then use to perform the sharding.

The entire concept of utilizing the LLM for the sharding of the PRD and architecture document is then rendered obsolete.

[ksylvan]

Then it also becomes easy to re-generate everything if you wanted to make an update to a sharded document. What do you think, Brian? @bmadcode

[bmadcode]

Yes, this is similar to what I was suggesting with the use of ========= start:filename ========== and ========= end:filename ========== sections within the doc - no worry about markdown confusion - just brute rip out between blocks.

We will need to:

• Update the templates with these marking indications
• Create a new shard script (nodejs and still some tech debt to introduce python scripts - or maybe just keep it simple to node) - or go generic linux style commands, but that always seems to run into issues with windows.
• Potentially need a migration script - there are lots of complaints when I release a change that is somewhat different, so maybe a script that can find all the sections and add the starter and ender tags.

Also - because of the way I set up v3 (in hind site i wish i didnt) - some sections get merged into a single file, instead of smaller granular files. Part of me wants to go back to the smaller granular files though (for example, instead of operational guidelines, have more granular smaller files based on or similar the markdown section names), as I was mistaken in thinking the LLM charges for each individual file it accesses.

What do you think @ksylvan?

[bmadcode]

Something like this in the template:

========= start:tech-stack.md ==========

Technology Stack Selections

{Collaboratively define the core technologies. Be specific about choices and versions where appropriate.}

• Primary Backend Language/Framework: {e.g., Python/FastAPI, Node.js/Express, Java/Spring Boot}
• Primary Frontend Language/Framework (if applicable): {e.g., TypeScript/React (Next.js), JavaScript/Vue.js}
• Database: {e.g., PostgreSQL, MongoDB, AWS DynamoDB}
• Key Libraries/Services (Backend): {e.g., Authentication (JWT, OAuth provider), ORM (SQLAlchemy), Caching (Redis)}
• Key Libraries/Services (Frontend, if applicable): {e.g., UI Component Library (Material-UI, Tailwind CSS + Headless UI), State Management (Redux, Zustand)}
• Deployment Platform/Environment: {e.g., Docker on AWS ECS, Vercel, Netlify, Kubernetes}
• Version Control System: {e.g., Git with GitHub/GitLab}

========= end:tech-stack.md ==========

[ksylvan]

Yeah, that's fine too, but it doesn't take care of one use case:

The idea of using Markdown comment syntax for the separators is this:

1. The LLM agent does not have to care about it. They do their job of just putting the information in the template and exporting it.

2. Then the generated PRD or architecture document becomes the source of truth for the sharded documents. If you change something that is in a section that is delimited by the markers,

<!-- ===start:tech-stack.md===  -->
# Technology Stack Selections
{Collaboratively define the core technologies. Be specific about choices and versions where appropriate.}

* Primary Backend Language/Framework: {e.g., Python/FastAPI, Node.js/Express, Java/Spring Boot}
* Primary Frontend Language/Framework (if applicable): {e.g., TypeScript/React (Next.js), JavaScript/Vue.js}
* Database: {e.g., PostgreSQL, MongoDB, AWS DynamoDB}
* Key Libraries/Services (Backend): {e.g., Authentication (JWT, OAuth provider), ORM (SQLAlchemy), Caching (Redis)}
* Key Libraries/Services (Frontend, if applicable): {e.g., UI Component Library (Material-UI, Tailwind CSS + Headless UI), State Management (Redux, Zustand)}
* Deployment Platform/Environment: {e.g., Docker on AWS ECS, Vercel, Netlify, Kubernetes}
* Version Control System: {e.g., Git with GitHub/GitLab}
 <!-- ===end:tech-stack.md===  -->

Then you can just re-run a command like node ./shard-document.js PRD.md to re-generate the documents.

[bmadcode]

Ah I think I understand - is it even worth adding the ======= filename separators than, or keep as is, but just update the task rule to wrap the content properly int he markdown comments?

Or maybe a combination of both?

[ksylvan]

I am suggesting that we only modify the templates to include the sharding metadata as comments.

This is separate from the system you already use to put multiple documents in one .txt file that the build-web-agent.js script uses so that you only have to upload a handful of documents to set up your Gem/CustomGPT. We don't have to change that.

Putting these metadata comments in the templates means that your PRD and architecture document already contain within them the instructions for how to shard them.

The only other change would be to remove the "shard document" task from the agents.

[bmadcode]

Got it! Ok let do it. Can you pr your script and template changes? If not I can try to replicate this weekend

…

On Wed, May 28, 2025, 10:39 PM Kayvan Sylvan ***@***.***> wrote: I am suggesting that we only modify the templates to include the sharding metadata as comments. This is separate from the system you already use to put multiple documents in one .txt file that the build-web-agent.js so that you only have to upload a handful of documents to set up your Gem/CustomGPT. We don't have to change that. Putting these metadata comments in the templates means that your PDR and architecture document already contain within them the instructions for how to shard them. The only other change would be to remove the "shard document" task from the agents. — Reply to this email directly, view it on GitHub <#111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BOTVGQL7BRSZZASQCS4I2HL3AZ6QFAVCNFSM6AAAAAB6B2YGU6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMZQGQ4TKOI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ksylvan]

I'll work on this. Might have something by today.

[ksylvan]

Hi Brian @bmadcode - Check this new tool out:

I just created it for this purpose: https://github.com/ksylvan/markdown-tree-parser

You can install it via npm:

$ npm install -g @kayvan/markdown-tree-parser 

added 57 packages in 2s

47 packages are looking for funding
  run `npm fund` for details

$ md-tree list ~/src/fabric-mcp/docs/architecture.md 

📋 Headings in architecture.md (32 total):

📁 Fabric MCP Server Architecture Document
  📄 Introduction / Preamble
  📄 Technical Summary
  📄 High-Level Overview
  📄 Architectural / Design Patterns Adopted
  📄 Component View
  📄 Project Structure
    📃 Key Directory Descriptions
    📃 Notes
  📄 API Reference
    📃 External APIs Consumed
      📃 Fabric REST API
    📃 Internal APIs Provided (MCP Tools)
  📄 Data Models
    📃 Core Application Entities / Domain Objects
    📃 API Payload Schemas (MCP Tools & Fabric API)
    📃 Database Schemas
  📄 Core Workflow / Sequence Diagrams
    📃 1. MCP Client: Tool Discovery ()
    📃 2. MCP Client: Execute Non-Streaming Pattern (e.g., )
    📃 3. MCP Client: Execute Streaming Pattern ( with )
    📃 4. Server Startup & Fabric API Connectivity Check
  📄 Definitive Tech Stack Selections
  📄 Infrastructure and Deployment Overview
  📄 Error Handling Strategy
  📄 Coding Standards
    📃 Detailed Language & Framework Conventions
      📃 Python Specifics
  📄 Overall Testing Strategy
  📄 Security Best Practices
  📄 Key Reference Documents
  📄 Change Log

For example:

$ md-tree extract ~/src/fabric-mcp/docs/architecture.md 'Coding Standards'

📄 Section "Coding Standards":

## Coding Standards

These standards are mandatory for all code generation by AI agents and human developers. Deviations are not permitted unless explicitly approved and documented as an exception in this section or a linked addendum.

* **Primary Language & Runtime:** Python >=3.11 with CPython (as per Definitive Tech Stack Selections).
* **Style Guide & Linter:**
  * **Tools:** `Ruff` for formatting and primary linting, `Pylint` for additional static analysis, `isort` for import sorting (often managed via Ruff). `Pyright` for static type checking in strict mode.
  * **Configuration:**
    * Ruff: Configured via `.ruff.toml` and/or `pyproject.toml`.
    * Pylint: Configured via `.pylintrc`.
    * isort: Configured in `pyproject.toml` (often via `[tool.ruff.lint.isort]`).
    * Pyright: Configured in `pyproject.toml` (under `[tool.pyright]`) for strict mode.

[ksylvan]

I'm rethinking my original idea of adding Markdown comment metadata...

I think we might be able to create a sharding config file that basically lists all the headings in the PRD.md or architecture.md documents that correspond to a sharded document and then a simple script can assemble the sharded documents using my md-tree tool.

[bmadcode]

This is awesome idea - I checked the repo - great idea. We can add a package.json - will make it easier for various scripts available also. Also going to add an install script (which just copies the bmad-agent folder to a project target).

[ksylvan]

Let me think on it a bit more and I'll come up with an excellent PR.

The concept involves a YAML configuration file, likely derived from the current sharding instructions.

Something like this:

bmad-agent/doc-sharding-config.yaml

# Document Sharding Configuration
# Maps destination documents to their source sections for automated sharding

version: "1.0"
description: "Configuration for sharding large project documents into smaller, focused files"

# Configuration for PRD document sharding
prd_sharding:
  source_document: "PRD.md"  # Can be overridden at runtime
  destination_prefix: "docs/"
  sections:
    - destination: "epic-{id}.md"
      source_sections:
        - "Epic {id}:*"  # Matches Epic headings with IDs
        - "**Epic {id}:**"  # Alternative Epic format
      description: "Individual Epic breakdown with user stories"
      pattern_type: "dynamic"  # Uses {id} placeholder
      
# Configuration for Architecture document sharding  
architecture_sharding:
  source_document: "architecture.md"  # Can be overridden at runtime
  destination_prefix: "docs/"
  sections:
    - destination: "api-reference.md"
      source_sections:
        - "API Reference"
        - "API Endpoints"
        - "Service Interfaces"
      description: "API documentation and endpoint specifications"
      
    - destination: "data-models.md"
      source_sections:
        - "Data Models"
        - "Database Schema"
        - "Entity Definitions"
      description: "Data structure and database schema definitions"
      
    - destination: "environment-vars.md"
      source_sections:
        - "Environment Variables Documentation"
        - "Configuration Settings"
        - "Deployment Parameters"
        - "Infrastructure and Deployment Overview" # Extract config details
      description: "Environment variables and configuration settings"
      
    - destination: "project-structure.md"
      source_sections:
        - "Project Structure"
        - "Detailed Frontend Directory Structure"
        - "Backend Repository Structure"
      description: "Code organization and directory structure"
      
    - destination: "tech-stack.md"
      source_sections:
        - "Technology Stack"
        - "Key Technologies"
        - "Libraries and Frameworks"
        - "Definitive Tech Stack Selections"
      description: "Technology choices and framework decisions"
      
    - destination: "operational-guidelines.md"
      source_sections:
        - "Coding Standards"
        - "Development Guidelines"
        - "Best Practices"
        - "Testing Strategy"
        - "Testing Decisions"
        - "QA Processes"
        - "Overall Testing Strategy"
        - "Error Handling Strategy"
        - "Security Best Practices"
      description: "Development practices and operational procedures"
      consolidation_note: "Organize under H3/H4 headings by source section"
      
    - destination: "component-view.md"
      source_sections:
        - "Component View"
        - "Architectural / Design Patterns Adopted"
      description: "System components and architectural patterns"
      
    - destination: "sequence-diagrams.md"
      source_sections:
        - "Core Workflow / Sequence Diagrams"
        - "Workflow Diagrams"
        - "Process Flow"
      description: "System workflows and process diagrams"
      
    - destination: "infra-deployment.md"
      source_sections:
        - "Infrastructure and Deployment Overview"
        - "Deployment Strategy"
        - "Cloud Infrastructure"
      description: "Infrastructure setup and deployment processes"
      
    - destination: "key-references.md"
      source_sections:
        - "Key Reference Documents"
        - "Related Documentation"
        - "External References"
      description: "Links to related documentation and external resources"

# Configuration for Frontend-specific document sharding
frontend_sharding:
  source_document: "front-end-architecture.md"  # Can be overridden at runtime
  destination_prefix: "docs/"
  sections:
    - destination: "front-end-project-structure.md"
      source_sections:
        - "Front-End Project Structure"
        - "Detailed Frontend Directory Structure"
        - "Component Organization"
      description: "Frontend code organization and file structure"
      
    - destination: "front-end-style-guide.md"
      source_sections:
        - "UI Style Guide"
        - "Brand Guidelines"
        - "Visual Design Specifications"
        - "Styling Approach"
      description: "UI styling guidelines and design system"
      
    - destination: "front-end-component-guide.md"
      source_sections:
        - "Component Library"
        - "Reusable UI Components Guide"
        - "Atomic Design Elements"
        - "Component Breakdown & Implementation Details"
      description: "Component library and reusable UI elements"
      
    - destination: "front-end-coding-standards.md"
      source_sections:
        - "Front-End Coding Standards"
        - "JavaScript/TypeScript Style"
        - "CSS Naming Conventions"
        - "Accessibility Best Practices"
      description: "Frontend-specific coding conventions and standards"
      
    - destination: "front-end-state-management.md"
      source_sections:
        - "State Management In-Depth"
        - "State Architecture"
        - "Data Flow Patterns"
      description: "State management patterns and data flow"
      
    - destination: "front-end-api-interaction.md"
      source_sections:
        - "API Interaction Layer"
        - "HTTP Client Setup"
        - "Data Fetching Patterns"
      description: "Frontend API communication and data fetching"
      
    - destination: "front-end-routing-strategy.md"
      source_sections:
        - "Routing Strategy"
        - "Navigation Patterns"
        - "Route Configuration"
      description: "Frontend routing and navigation implementation"
      
    - destination: "front-end-testing-strategy.md"
      source_sections:
        - "Frontend Testing Strategy"
        - "Component Testing"
        - "UI Testing Patterns"
      description: "Frontend testing approaches and methodologies"

# Global settings
global_settings:
  # Whether to update docs/index.md after sharding
  update_index: true
  
  # Index file configuration
  index_file: "docs/index.md"
  
  # Whether to preserve source document after sharding
  preserve_source: true
  
  # Default extraction behavior
  extraction_mode: "verbatim"  # Don't modify content during extraction
  
  # Whether to create destination directories if they don't exist
  create_directories: true
  
  # Backup settings
  create_backup: true
  backup_suffix: ".backup"
  
  # Content formatting
  preserve_markdown_formatting: true
  include_section_headers: true
  
  # Error handling
  skip_missing_sections: true
  warn_on_missing_sections: true

# Metadata for tracking
metadata:
  created_by: "bmad-agent"
  version: "1.0"
  last_updated: "2025-05-30"
  description: "Configuration for automated document sharding to improve developer documentation accessibility"

[bmadcode]

Looks like you are on it - thanks I had no time tonight! A few things I was thinking about that might be tricky (but you might already be considering.

Some sections result in multiple files with a pattern (such as the epics) - is there a way to genericize if there were another similar type of section in the future (if we gotta brute force it though, thats fine)

Right now the dumb way I set it up, multiple sections will go to a single file, which I hate in hindsight - I am fine with getting rid of this idea though (although we could make it an option) - But I think it would be much better to have each section being carved out go to its own files and we can just update the agent instructions (and I am working on a simplification for that also!) - so if it makes it easier, each section could just go to a file similar to the section name (or overridden in the template sharding instructions)

One other potential idea - more want an opinion though - the original idea for the sharding and I never got it fully implemented, was to actually explode the larger documents, and then have them have links to the smaller files, and not also retain the content in the larger files. The idea being - it would then be easier to make update for example to the source map, or coding standards or tech list - could just update the file directly.

Also - the idea is, a lot of these files should go away once MVP is done (or at least moved to an archive and ignored from the LLM in the future - but some files would still make sense to remain (source map, coding standards, maybe some others that will help the LLMs going forward in future endevours) - and a root contributing.md file being the entryway to actually finding these files for future human or ai contribution.

Anyways - this is all a bit more than what you got yourself into, just sharing what the longer term goal I never really got to was, but do see coming maybe with 3.2 or 3.3 - I think its a minor change, not requiring another version bump and migration IMO.

[ksylvan]

Right now the dumb way I set it up, multiple sections will go to a single file, which I hate in hindsight - I am fine with getting rid of this idea though (although we could make it an option) - But I think it would be much better to have each section being carved out go to its own files and we can just update the agent instructions (and I am working on a simplification for that also!) - so if it makes it easier, each section could just go to a file similar to the section name (or overridden in the template sharding instructions)

@bmadcode In terms of this question , Brian, you are the originator of the idea, so call your shot. Tell me how you'd like it, and with the md-tree tool we can easily make it work whichever way you prefer that would be most useful to future developers (both human and AI).

I really do like the idea of exploding the big doc into component parts and transforming the source document into just a table of contents placeholder.

🤔

One possible simplification could be to use the extract-all flag I set up in the tool to export all the level 2 headings, then we can use the toc tool (with some minor modification to create an index.md file which points to all the sharded mini-documents)

Proof-of-concept:

And generating the index.md

md-tree toc docs/architecture.md --output generated > generated/index.md

This results in a completely navigable tree like this:

$ tree generated 
generated
├── 01-introduction-preamble.md
├── 02-technical-summary.md
├── 03-high-level-overview.md
├── 04-architectural-design-patterns-adopted.md
├── 05-component-view.md
├── 06-project-structure.md
├── 07-api-reference.md
├── 08-data-models.md
├── 09-core-workflow-sequence-diagrams.md
├── 10-definitive-tech-stack-selections.md
├── 11-infrastructure-and-deployment-overview.md
├── 12-error-handling-strategy.md
├── 13-coding-standards.md
├── 14-overall-testing-strategy.md
├── 15-security-best-practices.md
├── 16-key-reference-documents.md
├── 17-change-log.md
└── index.md

1 directory, 18 files

[bmadcode]

Think I missed this before - I like this idea and also the stats!

[ksylvan]

And that is an old idea at this point.

The md-tree explode command sets up the index.md so no need to do it via the TOC command.

[ksylvan]

Another interesting capability of my tool, in case you like nerding out about stuff like this:

[ksylvan]

@bmadcode As of 1.2.1 (just published), md-tree can now properly explode a markdown file to a destination directory.

📚 md-tree v1.2.1 - Markdown Tree Parser CLI

Usage: md-tree <command> <file> [options]

Commands:
  list <file>                      List all headings in the file
  extract <file> <heading>         Extract a specific section by heading text
  extract-all <file> [level]       Extract all sections at level (default: 2)
  explode-doc <file> <output-dir>  Extract all level 2 sections and create index
  tree <file>                      Show the document structure as a tree
  search <file> <selector>         Search using CSS-like selectors
  stats <file>                     Show document statistics
  toc <file>                       Generate table of contents
  version                          Show version information
  help                             Show this help message

Options:
  --output, -o <dir>              Output directory for extracted files
  --level, -l <number>            Heading level to work with
  --format, -f <json|text>        Output format (default: text)
  --max-level <number>            Maximum heading level for TOC (default: 3)

Examples:
  md-tree list README.md
  md-tree extract README.md "Installation"
  md-tree extract-all README.md 2 --output ./sections
  md-tree explode-doc README.md ./exploded
  md-tree tree README.md
  md-tree search README.md "heading[depth=2]"
  md-tree stats README.md
  md-tree toc README.md --max-level 2

For more information, visit: https://github.com/ksylvan/markdown-tree-parser

For example, using my architecture.md file:

md-tree explode-doc architecture.md output_dir

The generated index.md properly references every file and they all start at heading level 1 (# Title)

Check it out! I think this is the simplest solution for us to use in managing BMAD-generated documents.

So the flow is:

• You chat with the agents, and going through the flow, you create PRD.md and architecture.md.
• You then run md-tree to explode your document (e.g. md-tree explode-doc docs/architecture.md docs/architecture to create a docs/architecture directrory with all the parts)
• Depending on your preference, you can even remove the architecture.md file at this point.
• In all other documents (the top level README or Contributing document) where you previously referenced docs/architecture.md, now you replace those references with docs/architecture/index.md

[ksylvan]

I fixed a few bugs in @kayvan/markdown-tree-parser

I renamed the sharding command to be calledexplode and implemented an assemble command to put it back together.

$ md-tree                               

📚 md-tree v1.3.2 - Markdown Tree Parser CLI

Usage: md-tree <command> <file> [options]

Commands:
  list <file>                   List all headings in the file
  extract <file> <heading>      Extract a specific section by heading text
  extract-all <file> [level]    Extract all sections at level (default: 2)
  explode <file> <output-dir>   Extract all level 2 sections and create index
  assemble <dir> <output-file>  Reassemble exploded document from directory
  tree <file>                   Show the document structure as a tree
  search <file> <selector>      Search using CSS-like selectors
  stats <file>                  Show document statistics
  toc <file>                    Generate table of contents
  version                       Show version information
  help                          Show this help message

Options:
  --output, -o <dir>            Output directory for extracted files
  --level, -l <number>          Heading level to work with
  --format, -f <json|text>      Output format (default: text)
  --max-level <number>          Maximum heading level for TOC (default: 3)

Examples:
  md-tree list README.md
  md-tree extract README.md "Installation"
  md-tree extract-all README.md 2 --output ./sections
  md-tree explode README.md ./exploded
  md-tree assemble ./exploded reassembled.md
  md-tree tree README.md
  md-tree search README.md "heading[depth=2]"
  md-tree stats README.md
  md-tree toc README.md --max-level 2

For more information, visit: https://github.com/ksylvan/markdown-tree-parser

Most important, if you run the explode and assemble commands, the operation perfectly recreates your original
large markdown file:

[bmadcode]

Md-Exploder! That is an epic name and potential logo, just sayin... 😀

…

On Fri, May 30, 2025, 3:21 PM Kayvan Sylvan ***@***.***> wrote: I fixed a few bugs in @kayvan/markdown-tree-parser I renamed the sharding command to be calledexplode and implemented an assemble command to put it back together. $ md-tree 📚 md-tree v1.3.2 - Markdown Tree Parser CLI Usage: md-tree <command> <file> [options] Commands: list <file> List all headings in the file extract <file> <heading> Extract a specific section by heading text extract-all <file> [level] Extract all sections at level (default: 2) explode <file> <output-dir> Extract all level 2 sections and create index assemble <dir> <output-file> Reassemble exploded document from directory tree <file> Show the document structure as a tree search <file> <selector> Search using CSS-like selectors stats <file> Show document statistics toc <file> Generate table of contents version Show version information help Show this help message Options: --output, -o <dir> Output directory for extracted files --level, -l <number> Heading level to work with --format, -f <json|text> Output format (default: text) --max-level <number> Maximum heading level for TOC (default: 3) Examples: md-tree list README.md md-tree extract README.md "Installation" md-tree extract-all README.md 2 --output ./sections md-tree explode README.md ./exploded md-tree assemble ./exploded reassembled.md md-tree tree README.md md-tree search README.md "heading[depth=2]" md-tree stats README.md md-tree toc README.md --max-level 2 For more information, visit: https://github.com/ksylvan/markdown-tree-parser Most important, if you run the explode and assemble commands, the operation perfectly recreates your original large markdown file: — Reply to this email directly, view it on GitHub <#111 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BOTVGQPWZGQP5WOM3XDB4X33BC4UFAVCNFSM6AAAAAB6B2YGU6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMZSGUYTINQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bmadcode]

Excited to get this all working tonight or this weekend. Also getting a lot of feedback about the current flow being too slow and long in some areas working with the agents. Have a good streamlining idea in mind without losing effectiveness. Combined with this, really gonna make things much easier and faster!

…

On Fri, May 30, 2025, 3:51 PM brian madison ***@***.***> wrote: Md-Exploder! That is an epic name and potential logo, just sayin... 😀 On Fri, May 30, 2025, 3:21 PM Kayvan Sylvan ***@***.***> wrote: > I fixed a few bugs in @kayvan/markdown-tree-parser > > I renamed the sharding command to be calledexplode and implemented an > assemble command to put it back together. > > $ md-tree > > 📚 md-tree v1.3.2 - Markdown Tree Parser CLI > > Usage: md-tree <command> <file> [options] > > Commands: > list <file> List all headings in the file > extract <file> <heading> Extract a specific section by heading text > extract-all <file> [level] Extract all sections at level (default: 2) > explode <file> <output-dir> Extract all level 2 sections and create index > assemble <dir> <output-file> Reassemble exploded document from directory > tree <file> Show the document structure as a tree > search <file> <selector> Search using CSS-like selectors > stats <file> Show document statistics > toc <file> Generate table of contents > version Show version information > help Show this help message > > Options: > --output, -o <dir> Output directory for extracted files > --level, -l <number> Heading level to work with > --format, -f <json|text> Output format (default: text) > --max-level <number> Maximum heading level for TOC (default: 3) > > Examples: > md-tree list README.md > md-tree extract README.md "Installation" > md-tree extract-all README.md 2 --output ./sections > md-tree explode README.md ./exploded > md-tree assemble ./exploded reassembled.md > md-tree tree README.md > md-tree search README.md "heading[depth=2]" > md-tree stats README.md > md-tree toc README.md --max-level 2 > > For more information, visit: https://github.com/ksylvan/markdown-tree-parser > > Most important, if you run the explode and assemble commands, the > operation perfectly recreates your original > large markdown file: > > — > Reply to this email directly, view it on GitHub > <#111 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/BOTVGQPWZGQP5WOM3XDB4X33BC4UFAVCNFSM6AAAAAB6B2YGU6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMZSGUYTINQ> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[ksylvan]

I'm done with md-tree for a bit.

The latest version is 1.4.0 and you can install it by

# Via npm
npm install -g @kayvan/markdown-tree-parser

# Via pnpm
pnpm install -g @kayvan/markdown-tree-parser

[bmadcode]

Awesome - I am very excited to try it this weekend - been a super busy work week not much chance to get to it so far. Will try to get it worked in and tested this weekend!

[icklers]

Hey guys, following your discussion I just want to show my appreciation for your dedication. I love your thought process of improving.

It was a bit annoying, honestly, to go through the compiled files. i love the md exploder ;)

This might also come in handy to create change validation reports and having a traceable history of the changes (should be optional, knowing it will increase the text files in the repos, but also can add an option to outsource it and store it on an S3 bucket).
For example: for each new version bump you create the validated change reports (Software, Infrastructure, ...)

let me know what you think - I can create a new issue for it, don't want to hijack this thread intentionally.

[ksylvan]

@icklers Thanks. @kayvan/markdown-tree-parser is now at version 1.4.1 (with the only functional change being the logo and some package upgrades).

I really like that the idea of this utility grew out of discussions with Brian @bmadcode around how to improve the document sharding and make it easier too, based on problems I ran into when trying to shard my large documents that contained diagrams or markdown code blocks.

[bmadcode]

Agree - and I have really enjoyed the collaboration @ksylvan so thank you again :) As the tree-parser is now stable, I am going to take a stab at making the various template improvements and give this integrated! I love having scripts that can run, takes so much variability out of the agents when certain things can be script based!

[ksylvan]

While the Gemini gem did a fine job (with my modified instructions) of generating a document that was ready to be sharded, it still needed post-processing with a bespoke Python script.

This new process is universally applicable, significantly simpler, and has produced a utility script suitable for numerous applications.

[bmadcode]

@ksylvan so to summarize the whole thread - no actual change of any kind of marking or indication needed in the templates - if I understand correctly, just need to ensure everything is at a Level ## in the MD to get pulled and added as a level 1? No more sharding templates, etc...?

I am about to fire it up and try it out, so Ill find out soon enough :)

[ksylvan]

Correct.

As long as the documents are created with proper headings (start at level 1 saying what the document is "# Product Requirements Document for My Awesome Project") and then have all the subsections at level 2 - and sub-subsections as needed, then the md-tree explode command will produce a navigable tree of properly formatted markdown files.

My convention would be to do this, for example:

$ md-tree explode docs/PRD.md docs/PRD

📚 Exploding 9 sections from PRD.md to docs/PRD:

✅ Written to docs/PRD/1-goal-objective-and-context.md
✅ Processing 1. Goal, Objective and Context → 1-goal-objective-and-context.md
✅ Written to docs/PRD/2-functional-requirements-mvp.md
✅ Processing 2. Functional Requirements (MVP) → 2-functional-requirements-mvp.md
✅ Written to docs/PRD/3-non-functional-requirements-mvp.md
✅ Processing 3. Non Functional Requirements (MVP) → 3-non-functional-requirements-mvp.md
✅ Written to docs/PRD/4-user-interaction-and-design-goals.md
✅ Processing 4. User Interaction and Design Goals → 4-user-interaction-and-design-goals.md
✅ Written to docs/PRD/5-technical-assumptions.md
✅ Processing 5. Technical Assumptions → 5-technical-assumptions.md
✅ Written to docs/PRD/epic-overview.md
✅ Processing Epic Overview → epic-overview.md
✅ Written to docs/PRD/6-key-reference-documents.md
✅ Processing 6. Key Reference Documents → 6-key-reference-documents.md
✅ Written to docs/PRD/7-out-of-scope-ideas-post-mvp.md
✅ Processing 7. Out of Scope Ideas Post MVP → 7-out-of-scope-ideas-post-mvp.md
✅ Written to docs/PRD/8-change-log.md
✅ Processing 8. Change Log → 8-change-log.md
✅ Written to docs/PRD/index.md
✅ Processing Table of Contents → index.md

✨ Document exploded to docs/PRD (10 files)

And navigating to one of the sub-documents:

And then in my README or other places where the PRD.md was mentioned, all I have to do is replace that with PRD/index.md

[ksylvan]

I can collaborate with you on a video if you'd like.

Fabric is basically a Swiss Army knife of crowd-sourced patterns and prompts and prompting patterns.

It's a CLI tool that works on macOS, Linux, and Windows. It makes all your AI providers available in one toolbox, and it allows you to run predefined prompts (called "patterns") and pipe in your own input.

I use it many, many times a day. For everything I'm coding, before every git commit, I run a pattern:

git diff --staged | summarize_git_diff

And that gives me a great git commit message.

Almost all my PRs are mostly created by the write_pull-request pattern with minor adjustments and corrections. That alone saves me 20-30 minutes every time I use it.

You can use it to summarize articles, write essays or letters, create reports, and even do some rudimentary web scraping.

Here's an example of its web scraping and then feeding it to a pattern:

$ fabric -u https://openai.com/sam-and-jony/ -p summarize_micro -m $MODEL_GROKAI

### ONE SENTENCE SUMMARY:
Sam Altman and Jony Ive introduce io, merging with OpenAI.

### MAIN POINTS:
- io, founded by Jony Ive, merges with OpenAI.
- Collaboration focuses on inspiring, empowering technology products.
- Jony Ive leads design across io and OpenAI.

### TAKEAWAYS:
- Innovative products aim to inspire and enable users.
- Deep integration enhances technology and design synergy.
- Shared vision drives optimistic, human-centric tools.

Here's another example:

yt -t 'https://youtu.be/E_QJ8j74U_0?si=UhlHWdrLLi8eaMXv' | fabric -p youtube_summary 

And that produced the output which follows:

BMAD Method V3: Revolutionary AI-Powered Development Workflow

Overview

This video introduces the BMAD Method V3, a revolutionary update to an AI-powered development methodology that helps developers build projects more efficiently using AI agents. The presenter demonstrates how the new BMAD agent orchestrates an entire AI agile team to take projects from idea to implementation while keeping development on track.

Key Sections

Why Vibe Coding Fails [00:00:42]

The presenter identifies common problems with traditional AI-assisted coding:

• Agents derail halfway through projects - As complexity increases, agents struggle to maintain coherence
• Resource waste - Without optimization and planning, credits and money get burned unnecessarily
• Poor information management - Agents put files in random places and forget previous decisions
• Mid-project changes are difficult - Recovering from major feature additions or changes is painful
• Lack of planning - "If you fail to plan, you are planning to fail"

Evolution of the BMAD Method [00:01:56]

V1 Origins:

• Started from "vibe coding" experiences
• Introduced cursor rules and documents to keep agents on track
• Created agile personas as custom modes
• Issues: Hardcoded agents, difficult to optimize and customize

V2 Improvements [00:02:35]:

• Moved planning work out of IDE to web platforms
• Leveraged deep research capabilities
• Key innovations: Checklists and templates
• Templates allowed defining PRD/architecture structure separately
• Agents trained to work section-by-section
• Still complex to set up across different platforms

V3 Revolutionary Updates [00:03:39]:

• Fully optimized with smaller agent files
• Separate files augment agent capabilities
• Works perfectly with Gemini Gems and ChatGPT
• Full customization capabilities
• New BMAD Agent - AI orchestration partner

The BMAD Agent [00:04:17]

The BMAD agent serves as:

• Method coach
• AI agile team organizer
• Dynamic adapter becoming the specific agent needed
• Coordinator for product manager, scrum master, and other roles

Full Agent Flow [00:04:44]

1. Brainstorming Phase [00:04:51]:

• Start with BMAD agent
• Work with Business Analyst Coach
• Create conceptual product brief
• Conduct deep research on competition
• Identify unique aspects

2. Product Management [00:05:28]:

• Product Manager creates PRD (Product Requirements Document)
• Defines high-level epics and stories
• Orders execution properly for developers

3. Design & Architecture [00:06:06]:

• UI/UX Design Architect handles frontend design
• Can create prompts for v0, Lovable, or other UI generators
• Application Architect defines source tree and tech decisions
• Considers user preferences for coding standards

4. Product Owner Review [00:08:32]:

• Runs masterful checklist
• Ensures all artifacts work together
• Updates everything for consistency

5. Development Preparation [00:09:08]:

• Scrum Master performs document sharding
• Creates detailed stories with tasks/subtasks
• Prepares artifacts for developers

V3 Customization Features [00:10:43]

Modular Components:

• Separate personas files
• Independent task definitions
• Customizable checklists
• Flexible templates
• Agent configuration files

Advanced Features [00:12:03]:

• Reflective elicitation and brainstorming
• Eight action options for critical review
• Self-reflection capabilities
• Alternative solution generation
• Design assumption challenges

Setup Instructions [00:13:19]

Simple Installation Process:

1. Clone the BMAD Method repository
2. Navigate to web-build-sample folder
3. Copy agent prompt (under 8,000 characters)
4. Create new Gemini Gem or ChatGPT
5. Paste prompt as instructions
6. Upload all other files as knowledge
7. Start using immediately

IDE Setup [00:32:00]:

• Copy BMAD agent folder to project
• Install scrum master and developer agents
• Follow create story → implement story workflow

Practical Demo [00:24:52]

The presenter demonstrates building a "Daily Digest" application that:

• Downloads Hacker News stories
• Creates AI-generated podcasts
• Sends daily podcasts via email
• Uses 80s retro CRT terminal aesthetic

Key Demo Points:

• One-shot UI generation using detailed prompts
• Automatic file organization following architecture
• Full working frontend from single prompt
• Seamless backend development workflow

Best Practices [00:33:31]

Story Development Workflow:

1. Scrum Master creates detailed story
2. Start new context window for Dev Agent
3. Implement story
4. New window for next story creation
5. Repeat cycle

Benefits of Sequential Development:

• Dev agent leaves notes for Scrum Master
• Learning from previous stories
• Continuous improvement
• Better context management

Conclusion

The BMAD Method V3 represents a significant advancement in AI-assisted development, offering a complete orchestration system that takes projects from initial idea through full implementation. The modular, customizable approach combined with the intelligent BMAD agent makes it accessible for developers of all experience levels while maintaining the structure needed for complex projects.

[icklers]

That's crazy. I love it! 🚀

[bmadcode]

Very nice! Yeah @ksylvan lets do a video on this one, maybe in a few weeks once I get the improvements wrapped up and the tutorial basic videos finally done!

How are you doing the command line yt command before piping it to fabric?

Well don't share that one too wide, I need the watch time LOL :)

[bmadcode]

ah never mind - found it in the docs, its an alias from fabric. So cool - using with MCP should be really nice.

[ksylvan]

Well don't share that one too wide, I need the watch time LOL :)

😆 Content Creators hate that alias. 😄

I look forward to it. Yeah, a few weeks from now should be good.

[bmadcode]

I am going to update the template to just make each epic a level 2 item instead of bolded items under the epics overview - basically I am going to optimize the template headings to support this process.

BTW @ksylvan found a minor bug in the repo when using extract instead of explode - I will try to add a fix for it later with a PR against your repo, want to help contribute back to your awesome new tool!

[ksylvan]

Great. What was the bug? I'm not surprised, since I spent most of my time fixing the explode->assemble process and making sure that's a no-op.

[ksylvan]

Please create an issue 🐛 (and optionally a PR) for me against the markdown-tree-parser 😄

[bmadcode]

Created it! TL;DR extract (not explode) sometimes grabs a few extra level 2 sections).

[ksylvan]

Fixed in v1.4.2 please update 😄

Thanks for the bug report: ksylvan/markdown-tree-parser#2

[bmadcode]

Awe you beat me to it 🤣

…

On Sat, May 31, 2025, 8:31 PM Kayvan Sylvan ***@***.***> wrote: Fixed in v1.4.2 please update 😄 Thanks for the bug report: ksylvan/markdown-tree-parser#2 <ksylvan/markdown-tree-parser#2> — Reply to this email directly, view it on GitHub <#111 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BOTVGQJDLA4EOMSKV56BZDL3BJJVNAVCNFSM6AAAAAB6B2YGU6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGMZTGE4TMMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fabb]

@ksylvan it would be great if i could run markdown-tree-parser with npx instead of having to install it globally. when i run npx markdown-tree-parser ... now, it says npm error could not determine executable to run. probably it is just some simple config in markdown-tree-parser's package.json.

[fabb]

nvm, i was using the wrong package, see #111 (comment)

[fabb]

oh-oh, there are 2 similarly named packages:

@bmadcode i think you are referencing the wrong one in v4 (npm install -g markdown-tree-parser is the wrong one as far as i can see)

cc @ksylvan

[ksylvan]

Yes, it needs to be @kayvan/markdown-tree-parser

[fabb]

i tried out exploding with markdown-tree-parser on an existing bmad v3 project where the architecture was already sharded partially. it resulted in some files like this one:

# 3. Component View

(Details are now in `docs/component-view.md`)

i think it would be great if the markdown-tree-parser explosion command would inline such files during explosion if it makes sense.

[ksylvan]

@fabb Please create an issue for me in the markdown-tree-parser repo and I'll eventually get to it.

[bmadcode]

Ah shoot - If I put the wrong one good catch! Ill check in a bit!