Advanced developer docs (#2569)

This adds advanced developer documentation for ECS tooling, which documents all steps of the generation pipeline, adds information on the ECS-OTel mapping process, adds pydoc to all functions. It also adds documentation on field re-use, subset and exclude filters.

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: mjwolf

USAGE.md

@@ -8,7 +8,7 @@ applies_to:
 
 # Generated artifacts [ecs-artifacts]
 
-ECS maintains a collection of artifacts which are generated based on the schema. Examples include Elasticsearch index templates, CSV, and Beats field mappings. The maintained artifacts can be found in the [ECS Github repo](https://github.com/elastic/ecs/blob/master/generated#artifacts-generated-from-ecs).
+ECS maintains a collection of artifacts which are generated based on the schema. Examples include Elasticsearch index templates, CSV, and Beats field mappings. The maintained artifacts can be found in the [ECS Github repo](https://github.com/elastic/ecs/blob/main/generated#artifacts-generated-from-ecs).
 
-Users can generate custom versions of these artifacts using the ECS project’s tooling. See the tooling [usage documentation](https://github.com/elastic/ecs/blob/master/USAGE.md) for more detail.
+Users can generate custom versions of these artifacts using the ECS project’s tooling. See the tooling [usage documentation](https://github.com/elastic/ecs/blob/main/USAGE.md) for more detail.
 

docs/reference/ecs-artifacts.md

@@ -22,7 +22,7 @@ Before you start a conversion, be sure that you understand the basics below.
 
 Make sure you understand the distinction between Core and Extended fields, as explained in the [Guidelines and Best Practices](/reference/ecs-guidelines.md).
 
-Core and Extended fields are documented in the [*ECS Field Reference*](/reference/ecs-field-reference.md) or, for a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/master/generated/csv/fields.csv).
+Core and Extended fields are documented in the [*ECS Field Reference*](/reference/ecs-field-reference.md) or, for a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/main/generated/csv/fields.csv).
 
 
 ### An approach to mapping an existing implementation [ecs-conv]

docs/reference/ecs-converting.md

@@ -16,7 +16,7 @@ ECS defines multiple groups of related fields. They are called "field sets". The
 
 All other field sets are defined as objects in Elasticsearch, under which all fields are defined.
 
-For a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/master/generated/csv/fields.csv).
+For a single page representation of all fields, please see the [generated CSV of fields](https://github.com/elastic/ecs/blob/main/generated/csv/fields.csv).
 
 
 ## Field sets [ecs-fieldsets]

docs/reference/ecs-field-reference.md

@@ -345,5 +345,5 @@ Like the other fields in the [related](/reference/ecs-related.md) field set, `re
 
 ## Mapping examples [ecs-user-usage-mappings]
 
-For examples of mapping events from various sources, you can look at [RFC 0007 in section Source Data](https://github.com/elastic/ecs/blob/master/rfcs/text/0007-multiple-users.md#source-data).
+For examples of mapping events from various sources, you can look at [RFC 0007 in section Source Data](https://github.com/elastic/ecs/blob/main/rfcs/text/0007-multiple-users.md#source-data).
 

docs/reference/ecs-user-usage.md

@@ -41,5 +41,5 @@ ECS is a permissive schema. If your events have additional data that cannot be m
 
 ECS improvements are released following [Semantic Versioning](https://semver.org/). Major ECS releases are planned to be aligned with major Elastic Stack releases.
 
-Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the [Contribution Guidelines](https://github.com/elastic/ecs/blob/master/CONTRIBUTING.md).
+Any feedback on the general structure, missing fields, or existing fields is appreciated. For contributions please read the [Contribution Guidelines](https://github.com/elastic/ecs/blob/main/CONTRIBUTING.md).
 

docs/reference/index.md

@@ -0,0 +1,159 @@
+# ECS Scripts Developer Documentation
+
+This directory contains developer-focused documentation for the ECS generation scripts.
+
+## Purpose
+
+The ECS repository includes a comprehensive toolchain for generating various artifacts from schema definitions. These developer guides explain:
+
+- **How each component works** internally
+- **Architecture and design decisions**
+- **How to make changes** and extend functionality
+- **Troubleshooting** common issues
+
+## Documentation Structure
+
+### Module-Specific Guides
+
+Each major generator module has its own detailed guide:
+
+- **[otel-integration.md](otel-integration.md)** - OpenTelemetry Semantic Conventions integration
+  - Validation of ECS ↔ OTel mappings
+  - Loading OTel definitions from GitHub
+  - Generating alignment summaries
+
+- **[markdown-generator.md](markdown-generator.md)** - Markdown documentation generation
+  - Rendering ECS schemas to human-readable docs
+  - Jinja2 template system and customization
+  - OTel alignment documentation
+  - Adding new page types
+
+- **[intermediate-files.md](intermediate-files.md)** - Intermediate file generation
+  - Flat and nested format representations
+  - Bridge between schema processing and artifact generation
+  - Top-level vs. reusable fieldsets
+  - Data structure reference
+
+- **[es-template.md](es-template.md)** - Elasticsearch template generation
+  - Composable vs. legacy template formats
+  - Field type mapping conversion
+  - Template customization and settings
+  - Installation and troubleshooting
+
+- **[csv-generator.md](csv-generator.md)** - CSV field reference generation
+  - Spreadsheet-compatible field export
+  - Column structure and multi-field handling
+  - Analysis and integration examples
+  - Usage in Excel, Google Sheets, databases
+
+- **[beats-generator.md](beats-generator.md)** - Beats field definition generation
+  - YAML field definitions for Elastic Beats
+  - Default field selection and allowlist
+  - Contextual naming and field groups
+  - Integration with Beat modules
+
+### Quick Reference
+
+For high-level usage information, see:
+- **[../../USAGE.md](../../USAGE.md)** - User guide for running the generators
+- **[../../CONTRIBUTING.md](../../CONTRIBUTING.md)** - Contribution guidelines
+
+## Scripts Overview
+
+The `scripts/` directory contains several key components:
+
+### Core Modules
+
+| Module | Purpose | Documentation |
+|--------|---------|---------------|
+| `generator.py` | **Main entry point** - orchestrates complete pipeline | Comprehensive docstrings in file |
+| `generators/otel.py` | OTel integration and validation | [otel-integration.md](otel-integration.md) |
+| `generators/markdown_fields.py` | Markdown documentation generation | [markdown-generator.md](markdown-generator.md) |
+| `generators/intermediate_files.py` | Intermediate format generation | [intermediate-files.md](intermediate-files.md) |
+| `generators/es_template.py` | Elasticsearch template generation | [es-template.md](es-template.md) |
+| `generators/csv_generator.py` | CSV field reference export | [csv-generator.md](csv-generator.md) |
+| `generators/beats.py` | Beats field definition generation | [beats-generator.md](beats-generator.md) |
+| `generators/ecs_helpers.py` | Shared utility functions | See docstrings in file |
+
+### Schema Processing
+
+The schema processing pipeline transforms YAML schema definitions through multiple stages. See [schema-pipeline.md](schema-pipeline.md) for complete pipeline documentation.
+
+| Module | Purpose | Documentation |
+|--------|---------|---------------|
+| **Pipeline Overview** | Complete schema processing flow | **[schema-pipeline.md](schema-pipeline.md)** |
+| `schema/loader.py` | Load and parse YAML schemas → nested structure | [schema-pipeline.md#1-loaderpy---schema-loading](schema-pipeline.md#1-loaderpy---schema-loading) |
+| `schema/cleaner.py` | Validate, normalize, apply defaults | [schema-pipeline.md#2-cleanerpy---validation--normalization](schema-pipeline.md#2-cleanerpy---validation--normalization) |
+| `schema/finalizer.py` | Perform field reuse, calculate names | [schema-pipeline.md#3-finalizerpy---field-reuse--name-calculation](schema-pipeline.md#3-finalizerpy---field-reuse--name-calculation) |
+| `schema/visitor.py` | Traverse field hierarchies (visitor pattern) | [schema-pipeline.md#visitorpy---field-traversal](schema-pipeline.md#visitorpy---field-traversal) |
+| `schema/subset_filter.py` | Filter to include only specified fields | [schema-pipeline.md#4-subset_filterpy---subset-filtering-optional](schema-pipeline.md#4-subset_filterpy---subset-filtering-optional) |
+| `schema/exclude_filter.py` | Explicitly remove specified fields | [schema-pipeline.md#5-exclude_filterpy---exclude-filtering-optional](schema-pipeline.md#5-exclude_filterpy---exclude-filtering-optional) |
+
+### Types
+
+| Module | Purpose |
+|--------|---------|
+| `ecs_types/schema_fields.py` | Core ECS type definitions |
+| `ecs_types/otel_types.py` | OTel-specific types |
+
+## Getting Started
+
+If you're new to the ECS generator codebase:
+
+1. **Start with the main orchestrator**: Read `generator.py` docstrings to understand the pipeline
+2. **Understand schema processing**: Read [schema-pipeline.md](schema-pipeline.md)
+3. **Pick a generator**: Choose a specific generator that interests you
+4. **Read its documentation**: Start with the module-specific guide
+5. **Explore the code**: Read the source with the guide as reference
+6. **Run it**: Try generating artifacts to see it in action
+
+### Quick Command Reference
+
+```bash
+# Standard generation (from local schemas)
+python scripts/generator.py --semconv-version v1.24.0
+
+# From specific git version
+python scripts/generator.py --ref v8.10.0 --semconv-version v1.24.0
+
+# With custom schemas
+python scripts/generator.py --include custom/schemas/ --semconv-version v1.24.0
+
+# Generate subset only
+python scripts/generator.py --subset schemas/subsets/minimal.yml --semconv-version v1.24.0
+
+# Strict validation mode
+python scripts/generator.py --strict --semconv-version v1.24.0
+
+# Intermediate files only (fast iteration)
+python scripts/generator.py --intermediate-only --semconv-version v1.24.0
+```
+
+See `generator.py` docstrings for complete argument documentation.
+
+## Contributing Documentation
+
+When adding or modifying generator code:
+
+1. **Update docstrings**: Add comprehensive Python docstrings to all functions and classes
+2. **Update/create guide**: Ensure a markdown guide exists explaining the component
+3. **Update this README**: Add links to new documentation
+4. **Include examples**: Show practical usage examples
+5. **Document edge cases**: Explain tricky parts and gotchas
+
+### Documentation Standards
+
+- **Python docstrings**: Use Google-style docstrings with Args, Returns, Raises, Examples
+- **Markdown guides**: Include Overview, Architecture, Usage Examples, Troubleshooting
+- **Code examples**: Should be runnable (or clearly marked as pseudocode)
+- **Diagrams**: Use ASCII/Unicode diagrams for flow visualization
+- **Tables**: Use markdown tables for structured comparisons
+
+## Questions?
+
+For questions about:
+- **Using the tools**: See [USAGE.md](../../USAGE.md) or ask in the [Elastic community forums](https://discuss.elastic.co/)
+- **Contributing**: See [CONTRIBUTING.md](../../CONTRIBUTING.md)
+- **Architecture**: Read the relevant module guide in this directory
+- **Bugs**: [Open an issue](https://github.com/elastic/ecs/issues)
+