align to llms.txt (#27)

Author: mensfeld

CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 0.10.0 (2025-10-27)
+- [Feature] **llms.txt Specification Compliance** - Updated output format to fully comply with the llms.txt specification from llmstxt.org.
+  - **Metadata Format**: Metadata now appears within the description field using parentheses and comma separators: `- [title](url): description (tokens:450, updated:2025-10-13, priority:high)`
+  - **Optional Descriptions**: Parser now correctly handles links without descriptions: `- [title](url)` per spec
+  - **Multi-Section Support**: Documents automatically organized into `Documentation`, `Examples`, and `Optional` sections based on priority
+  - **Body Content Support**: Added optional `body` config parameter for custom content between description and sections
+  - Priority-based categorization: 1-3 → Documentation, 4-5 → Examples, 6-7 → Optional
+  - Empty sections are automatically omitted from output
+  - Updated parser regex from `/^[-*]\s*\[([^\]]+)\]\(([^)]+)\):\s*(.*)$/m` to `/^[-*]\s*\[([^\]]+)\]\(([^)]+)\)(?::\s*([^\n]*))?$/` to make descriptions optional
+  - Fixed multiline regex greedy matching issue that was capturing only one link per section
+- [Test] Added comprehensive test suite for spec compliance (8 new parser tests, 7 new generator tests)
+- [Docs] Updated README with multi-section organization examples and body content usage
+- **Breaking Change**: Metadata format has changed from `tokens:450 updated:2025-10-13` to `(tokens:450, updated:2025-10-13)` for spec compliance
+
 ## 0.9.4 (2025-10-27)
 - [Feature] **Auto-Exclude Hidden Directories** - Hidden directories (starting with `.`) are now automatically excluded by default to prevent noise from `.git`, `.lint`, `.github`, etc.
   - Adds `include_hidden: false` as default behavior

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    llm-docs-builder (0.9.4)
+    llm-docs-builder (0.10.0)
       zeitwerk (~> 2.6)
 
 GEM

README.md

@@ -109,6 +109,7 @@ docs: ./docs
 base_url: https://myproject.io
 title: My Project
 description: Brief description
+body: Optional body content between description and sections
 output: llms.txt
 suffix: .llm
 verbose: false
@@ -289,9 +290,9 @@ Generate enriched llms.txt files with token counts, timestamps, and priority lab
 
 **Enhanced llms.txt (with metadata enabled):**
 ```markdown
-- [Getting Started](https://myproject.io/docs/Getting-Started.md) tokens:450 updated:2025-10-13 priority:high
-- [Configuration](https://myproject.io/docs/Configuration.md) tokens:2800 updated:2025-10-12 priority:high
-- [Advanced Topics](https://myproject.io/docs/Advanced.md) tokens:5200 updated:2025-09-15 priority:medium
+- [Getting Started](https://myproject.io/docs/Getting-Started.md): Quick start guide (tokens:450, updated:2025-10-13, priority:high)
+- [Configuration](https://myproject.io/docs/Configuration.md): Configuration options (tokens:2800, updated:2025-10-12, priority:high)
+- [Advanced Topics](https://myproject.io/docs/Advanced.md): Deep dive topics (tokens:5200, updated:2025-09-15, priority:medium)
 ```
 
 **Benefits:**
@@ -309,6 +310,68 @@ include_priority: true      # Show priority labels (high/medium/low)
 calculate_compression: true # Show compression ratios (slower, requires transformation)
 ```
 
+**Note:** Metadata is formatted according to the llms.txt specification, appearing within the description field using parentheses and comma separators for spec compliance.
+
+### Multi-Section Organization
+
+Documents are automatically organized into multiple sections based on priority, following the llms.txt specification:
+
+**Priority-based categorization:**
+- **Documentation** (priority 1-3): Essential docs like README, getting started guides, user guides
+- **Examples** (priority 4-5): Tutorials and example files
+- **Optional** (priority 6-7): Advanced topics and reference documentation
+
+**Example output:**
+```markdown
+# My Project
+
+> Project description
+
+## Documentation
+
+- [README](README.md): Main documentation
+- [Getting Started](getting-started.md): Quick start guide
+
+## Examples
+
+- [Basic Tutorial](tutorial.md): Step-by-step tutorial
+- [Code Examples](examples.md): Example code
+
+## Optional
+
+- [Advanced Topics](advanced.md): Deep dive into advanced features
+- [API Reference](reference.md): Complete API reference
+```
+
+Empty sections are automatically omitted. The "Optional" section aligns with the llms.txt spec for marking secondary content that can be skipped when context windows are limited.
+
+### Body Content
+
+Add custom body content between the description and documentation sections:
+
+```yaml
+# llm-docs-builder.yml
+title: My Project
+description: Brief description
+body: |
+  This framework is built on Ruby and focuses on performance.
+  Key concepts: streaming, batching, and parallel processing.
+docs: ./docs
+```
+
+This produces:
+```markdown
+# My Project
+
+> Brief description
+
+This framework is built on Ruby and focuses on performance.
+Key concepts: streaming, batching, and parallel processing.
+
+## Documentation
+...
+```
+
 ## Advanced Compression Options
 
 All compression features can be used individually for fine-grained control:

lib/llm_docs_builder/config.rb

@@ -61,6 +61,7 @@ def merge_with_options(options)
         base_url: options[:base_url] || self['base_url'],
         title: options[:title] || self['title'],
         description: options[:description] || self['description'],
+        body: options[:body] || self['body'],
         output: options[:output] || self['output'] || 'llms.txt',
         convert_urls: if options.key?(:convert_urls)
                         options[:convert_urls]

lib/llm_docs_builder/generator.rb

@@ -210,7 +210,11 @@ def apply_transformations(content, file_path)
 
     # Constructs llms.txt content from analyzed documentation files
     #
-    # Combines title, description, and documentation links into formatted output
+    # Combines title, description, body content, and documentation links into formatted output.
+    # Organizes documents into sections based on priority:
+    # - Priority 1-3: Documentation (essential docs like README, getting started)
+    # - Priority 4-5: Examples (tutorials, example files)
+    # - Priority 6-7: Optional (advanced topics, reference docs)
     #
     # @param docs [Array<Hash>] analyzed file metadata
     # @return [String] formatted llms.txt content
@@ -224,31 +228,60 @@ def build_llms_txt(docs)
       content << "> #{description}" if description
       content << ''
 
-      if docs.any?
-        content << '## Documentation'
+      # Add optional body content
+      if options[:body] && !options[:body].empty?
+        content << options[:body]
         content << ''
+      end
 
-        docs.each do |doc|
-          url = build_url(doc[:path])
-          line = if doc[:description] && !doc[:description].empty?
-                   "- [#{doc[:title]}](#{url}): #{doc[:description]}"
-                 else
-                   "- [#{doc[:title]}](#{url})"
-                 end
-
-          # Append metadata if enabled
-          if options[:include_metadata]
-            metadata_parts = []
-            metadata_parts << "tokens:#{doc[:tokens]}" if doc[:tokens]
-            metadata_parts << "compression:#{doc[:compression]}" if doc[:compression]
-            metadata_parts << "updated:#{doc[:updated]}" if doc[:updated]
-            metadata_parts << priority_label(doc[:priority]) if options[:include_priority]
-
-            line += " #{metadata_parts.join(' ')}" unless metadata_parts.empty?
+      if docs.any?
+        # Categorize docs by priority into sections
+        sections = {
+          'Documentation' => docs.select { |d| d[:priority] <= 3 },
+          'Examples' => docs.select { |d| d[:priority] >= 4 && d[:priority] <= 5 },
+          'Optional' => docs.select { |d| d[:priority] >= 6 }
+        }
+
+        # Build each section (skip empty ones)
+        sections.each do |section_name, section_docs|
+          next if section_docs.empty?
+
+          content << "## #{section_name}"
+          content << ''
+
+          section_docs.each do |doc|
+            url = build_url(doc[:path])
+
+            # Build metadata string if enabled
+            metadata_str = nil
+            if options[:include_metadata]
+              metadata_parts = []
+              metadata_parts << "tokens:#{doc[:tokens]}" if doc[:tokens]
+              metadata_parts << "compression:#{doc[:compression]}" if doc[:compression]
+              metadata_parts << "updated:#{doc[:updated]}" if doc[:updated]
+              metadata_parts << priority_label(doc[:priority]) if options[:include_priority]
+
+              metadata_str = "(#{metadata_parts.join(', ')})" unless metadata_parts.empty?
+            end
+
+            # Build line according to spec: - [title](url): description (metadata)
+            line = if doc[:description] && !doc[:description].empty?
+                     base = "- [#{doc[:title]}](#{url}): #{doc[:description]}"
+                     metadata_str ? "#{base} #{metadata_str}" : base
+                   else
+                     # No description: - [title](url) (metadata)
+                     base = "- [#{doc[:title]}](#{url})"
+                     metadata_str ? "#{base}: #{metadata_str}" : base
+                   end
+
+            content << line
           end
 
-          content << line
+          content << ''
         end
+
+        # Remove trailing empty line
+        content.pop if content.last == ''
       end
 
       "#{content.join("\n")}\n"

lib/llm_docs_builder/parser.rb

@@ -85,19 +85,21 @@ def save_section(sections, section_name, content)
 
     # Extracts markdown links from section content into structured format
     #
-    # Scans for markdown list items with links and descriptions. Returns raw content
+    # Scans for markdown list items with links and optional descriptions. Returns raw content
     # if no links are found in the expected format.
     #
     # @param content [String] raw section content
     # @return [Array<Hash>, String] array of link hashes or raw content if no links found
     def parse_section_content(content)
       links = []
 
-      content.scan(/^[-*]\s*\[([^\]]+)\]\(([^)]+)\):\s*(.*)$/m) do |title, url, description|
+      # Updated regex: description is optional (non-capturing group with ?)
+      # Use [^\n]* instead of .* to avoid matching across lines
+      content.scan(/^[-*]\s*\[([^\]]+)\]\(([^)]+)\)(?::\s*([^\n]*))?$/) do |title, url, description|
         links << {
           title: title,
           url: url,
-          description: description.strip
+          description: description&.strip || ''
         }
       end
 

lib/llm_docs_builder/version.rb

@@ -2,5 +2,5 @@
 
 module LlmDocsBuilder
   # Current version of the LlmDocsBuilder gem
-  VERSION = '0.9.4'
+  VERSION = '0.10.0'
 end

spec/config_spec.rb

@@ -51,6 +51,34 @@
     end
   end
 
+  describe '#merge_with_options' do
+    it 'includes body parameter from config file' do
+      config_file = Tempfile.new(['config', '.yml'])
+      config_file.write(YAML.dump({ 'body' => 'Custom body content from config' }))
+      config_file.close
+
+      config = described_class.new(config_file.path)
+      merged = config.merge_with_options({})
+
+      expect(merged[:body]).to eq('Custom body content from config')
+
+      config_file.unlink
+    end
+
+    it 'allows CLI options to override body parameter' do
+      config_file = Tempfile.new(['config', '.yml'])
+      config_file.write(YAML.dump({ 'body' => 'Config body' }))
+      config_file.close
+
+      config = described_class.new(config_file.path)
+      merged = config.merge_with_options({ body: 'CLI body override' })
+
+      expect(merged[:body]).to eq('CLI body override')
+
+      config_file.unlink
+    end
+  end
+
   describe 'error handling' do
     it 'raises GenerationError for invalid YAML syntax' do
       config_file = Tempfile.new(['config', '.yml'])