Skip to content

Commit 1d25c03

Browse files
theletterfchalin
theletterf
authored and
chalin
committed
Add project proposals (new and ongoing)
1 parent 5c4585a commit 1d25c03

8 files changed

+1116
-1
lines changed

‎projects/collector-docs-refactoring.md‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
# Collector Documentation Refactoring
2+
3+
## Background and description
4+
5+
The OpenTelemetry Collector is a critical component of the OpenTelemetry ecosystem, serving as a vendor-agnostic way to receive, process, and export telemetry data. The Collector documentation on [opentelemetry.io](https://opentelemetry.io/docs/collector/) is currently undergoing a significant refactoring effort to improve organization, clarity, and completeness.
6+
7+
This project coordinates the ongoing refactoring of Collector documentation, which is being tracked through the `sig:collector:refactor` label and organized into multiple phases.
8+
9+
**Note**: This project is already underway with active issues and a milestone. This document serves to formalize the project scope and track progress within the community projects framework.
10+
11+
### Current challenges
12+
13+
The current Collector documentation has several issues being addressed:
14+
15+
- **Information architecture needs improvement**: Pages are not organized optimally for different user journeys (installation, deployment, configuration, extension).
16+
- **Large monolithic pages**: Some pages (like installation) need to be split into focused child pages.
17+
- **Inconsistent examples**: Examples sometimes use core vs. contrib inconsistently.
18+
- **Missing operational guidance**: Best practices, troubleshooting, and disaster recovery documentation is incomplete.
19+
- **Copy editing needed**: Many pages need style guide compliance, grammar fixes, and clarity improvements.
20+
- **Cross-reference maintenance**: Page renames require updating cross-references throughout the documentation.
21+
22+
**Current structure verified in repository (`content/en/docs/collector/`):**
23+
24+
```
25+
collector/
26+
├── _index.md # Landing page
27+
├── architecture.md # Architecture overview
28+
├── benchmarks.md # Performance benchmarks
29+
├── building/ # ✅ Custom collector building
30+
│ ├── _index.md
31+
│ ├── authenticator-extension.md
32+
│ ├── connector/
33+
│ └── receiver.md
34+
├── components/ # ✅ Component documentation
35+
│ ├── connector.md
36+
│ ├── exporter.md
37+
│ ├── extension.md
38+
│ ├── processor.md
39+
│ └── receiver.md
40+
├── configuration.md # Configuration reference
41+
├── custom-collector.md # Custom collector guide
42+
├── deployment/ # ✅ Deployment patterns
43+
│ ├── _index.md
44+
│ ├── agent.md
45+
│ ├── gateway/
46+
│ └── no-collector.md
47+
├── distributions.md # Distribution information
48+
├── installation.md # Installation (needs splitting)
49+
├── internal-telemetry.md # Internal metrics
50+
├── management.md # OpAMP management
51+
├── quick-start.md # Quick start guide
52+
├── resiliency.md # Resiliency features
53+
├── scaling.md # Scaling guide
54+
├── transforming-telemetry.md
55+
└── troubleshooting.md # Troubleshooting guide
56+
```
57+
58+
**Progress on refactoring:**
59+
- ✅ `building/` section exists with custom component guides
60+
- ✅ `deployment/` section has agent, gateway, no-collector patterns
61+
- ✅ `components/` section documents all component types
62+
- ⚠️ `installation.md` still monolithic (issue #8353)
63+
- ⚠️ Troubleshooting exists but needs expansion
64+
65+
### Goals, objectives, and requirements
66+
67+
The goal of this project is to:
68+
69+
1. **Restructure information architecture**: Create a logical hierarchy (Install, Deploy, Extend sections).
70+
2. **Split large pages**: Break monolithic pages into focused, scannable content.
71+
3. **Standardize examples**: Ensure consistent use of Collector distributions in examples.
72+
4. **Add operational content**: Include best practices, troubleshooting, and scaling documentation.
73+
5. **Copy edit all pages**: Ensure style guide compliance and clear writing.
74+
6. **Maintain cross-references**: Update all links when pages are renamed or moved.
75+
76+
## Deliverables
77+
78+
### Phase 1 (otelcol-phase-1) - Current
79+
80+
Focus on moving existing content to conform to the new information architecture:
81+
82+
- Rename and split Collector installation page.
83+
- Rename Collector deployment pages and create new child pages.
84+
- Create new "Extend the Collector" section.
85+
- Copy edit installation, deployment, and extension pages.
86+
- Update cross-references throughout documentation.
87+
88+
### Phase 2 (otelcol-phase-2)
89+
90+
Focus on writing and editing content (details TBD based on Phase 1 completion).
91+
92+
### Phase 3 (otelcol-phase-3)
93+
94+
Gap analysis of the new documentation set:
95+
96+
- Audit documentation for missing or incorrect content.
97+
- Identify which examples to add first.
98+
- Use AI, user interviews, and community feedback for analysis.
99+
- Address disaster recovery recommendations.
100+
- Follow up on deployment documentation improvements.
101+
102+
## Timeline
103+
104+
- **Phase 1**: Due December 31, 2025.
105+
- **Phase 2**: TBD.
106+
- **Phase 3**: Due March 20, 2026.
107+
108+
## Labels
109+
110+
- `sig:collector`
111+
- `sig:collector:refactor`
112+
- `docs:information-architecture`
113+
- `help wanted`
114+
- `good first issue` (for some tasks)
115+
116+
## Related issues
117+
118+
### Phase 1 issues (otelcol-phase-1 milestone)
119+
120+
- [#8353 - Rename and split up Collector installation page](https://github.com/open-telemetry/opentelemetry.io/issues/8353) (open)
121+
- [#8354 - Copy edit the Collector installation page](https://github.com/open-telemetry/opentelemetry.io/issues/8354) (open)
122+
- [#8355 - Rename Collector deployment pages and create new child pages](https://github.com/open-telemetry/opentelemetry.io/issues/8355) (open)
123+
- [#8356 - Copy edit the Collector deployment landing, no collector, and agent pages](https://github.com/open-telemetry/opentelemetry.io/issues/8356) (closed)
124+
- [#8358 - Copy edit the Collector deployment gateway page](https://github.com/open-telemetry/opentelemetry.io/issues/8358) (closed)
125+
- [#8360 - Create new "Extend the Collector" section and rename child pages](https://github.com/open-telemetry/opentelemetry.io/issues/8360) (open)
126+
- [#8361 - Copy edit the Building a custom Collector page](https://github.com/open-telemetry/opentelemetry.io/issues/8361) (open)
127+
- [#8362 - Copy edit the custom component pages](https://github.com/open-telemetry/opentelemetry.io/issues/8362) (open)
128+
129+
### Phase 3 issues (otelcol-phase-3 milestone)
130+
131+
- [#5932 - Disaster recovery recommendations for Daemonset deployments](https://github.com/open-telemetry/opentelemetry.io/issues/5932) (open)
132+
- [#2692 - Follow up on collector deployment](https://github.com/open-telemetry/opentelemetry.io/issues/2692) (open)
133+
134+
### Other related issues
135+
136+
- [#7268 - Add example for validating internal telemetry connection](https://github.com/open-telemetry/opentelemetry.io/issues/7268) (open)
137+
- [#3699 - Benchmarks page clarity](https://github.com/open-telemetry/opentelemetry.io/issues/3699) (open)
138+
139+
## Project Board
140+
141+
The project is tracked via GitHub milestones:
142+
143+
- [otelcol-phase-1](https://github.com/open-telemetry/opentelemetry.io/milestone/20)
144+
- [otelcol-phase-3](https://github.com/open-telemetry/opentelemetry.io/milestone/22)
145+
146+
## SIG Meetings and Other Info
147+
148+
This project is coordinated through SIG Communications and SIG Collector meetings.
149+
150+
- **Slack channel**: `#otel-comms`
151+
- **Meeting notes**: To be linked upon project start.
152+

‎projects/docs-automation-cicd.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Documentation Automation and CI/CD Enhancement
2+
3+
## Background and description
4+
5+
The OpenTelemetry documentation website relies on various automated processes for building, validating, and maintaining content. However, several manual processes remain, and existing automation has limitations that cause maintenance burden and occasional build failures.
6+
7+
This project aims to enhance documentation automation, improve CI/CD pipelines, and implement systems for tracking documentation freshness and detecting outdated content.
8+
9+
### Current challenges
10+
11+
The current documentation automation has several gaps:
12+
13+
- **Manual Collector metrics maintenance**: The list of Collector internal metrics on the website is manually maintained and can become out of sync with the actual metrics generated by `mdatagen`. This leads to incomplete or inaccurate documentation.
14+
- **Version update automation issues**: The automated version update process for Collector releases can break builds when patch releases don't include all components (e.g., Collector Builder or Supervisor). The `cascade` field in Hugo front matter causes issues with partial releases.
15+
- **No automated outdated content detection**: There's no systematic way to identify documentation that has become stale or needs updating. Content can remain outdated for extended periods without maintainer awareness.
16+
- **Missing mdatagen integration**: The `mdatagen` tool generates documentation for Collector components, but this isn't integrated into the website documentation pipeline.
17+
- **Limited pre-commit checks**: Contributors may submit PRs that fail CI checks because there are no local pre-commit hooks to catch common issues before pushing.
18+
19+
**Existing automation verified in repository (`scripts/`):**
20+
21+
```
22+
scripts/
23+
├── auto-update/ # Version update scripts
24+
│ ├── all-versions.sh # Updates all component versions
25+
│ └── version-in-file.sh # Updates versions in specific files
26+
├── git/
27+
│ └── pre-commit.sh # Basic pre-commit (only runs `npm run fix:format:staged`)
28+
├── _md-rules/ # Markdown validation rules
29+
│ ├── trim-code-blank-lines.js
30+
│ ├── unindent-code-blocks.js
31+
│ └── validate-links.js
32+
├── check-build-log.sh # Build log validation
33+
├── check-i18n.sh # Localization drift detection
34+
├── check-registry-urls # Registry URL validation
35+
├── registry-scanner/ # Registry scanning tools
36+
├── stats/ # Documentation statistics
37+
└── content-modules/ # Content module processing
38+
```
39+
40+
**Gap analysis:**
41+
- ✅ Version update scripts exist but have issues with patch releases
42+
- ⚠️ Pre-commit hook exists but is minimal (only formatting)
43+
- ⚠️ Markdown rules exist but not integrated with pre-commit
44+
- ❌ No mdatagen integration
45+
- ❌ No automated freshness tracking
46+
- ❌ No husky setup for comprehensive pre-commit hooks
47+
48+
If these challenges are not addressed:
49+
50+
- Documentation will continue to drift from actual software behavior.
51+
- Maintainers will spend excessive time on manual updates and build fixes.
52+
- Users will encounter inaccurate or outdated information.
53+
- Contributor experience will suffer from slow feedback loops.
54+
55+
### Goals, objectives, and requirements
56+
57+
The goal of this project is to:
58+
59+
1. **Automate Collector metrics documentation**: Integrate mdatagen output into the website.
60+
2. **Fix version update automation**: Handle patch releases correctly without breaking builds.
61+
3. **Detect outdated content**: Implement systems to identify stale documentation.
62+
4. **Improve contributor experience**: Add pre-commit hooks and local validation tools.
63+
5. **Track documentation freshness**: Create visibility into documentation age and update needs.
64+
65+
**Motivations for starting now**:
66+
67+
- The Collector documentation refactoring project (sig:collector:refactor) would benefit from automated tooling.
68+
- Recent build failures from patch releases highlight the urgency of fixing version automation.
69+
- Community feedback indicates that outdated content is a significant problem.
70+
71+
## Deliverables
72+
73+
### mdatagen integration
74+
75+
- **Automated metrics documentation**: Pipeline to extract and publish Collector component metrics from mdatagen.
76+
- **Component documentation sync**: Automated sync of component READMEs to website.
77+
- **Stability level tracking**: Automated tracking and display of component stability levels.
78+
79+
### Version automation improvements
80+
81+
- **Patch release handling**: Fix the cascade version update mechanism to handle partial releases.
82+
- **Component-specific versioning**: Separate version tracking for Collector core, contrib, Builder, and Supervisor.
83+
- **Release notes integration**: Automated linking to release notes for each version.
84+
85+
### Outdated content detection
86+
87+
- **Content age tracking**: System to track when content was last updated.
88+
- **Drift detection alerts**: Automated notifications when content hasn't been updated within a threshold.
89+
- **External link validation**: Regular checks for broken external links.
90+
- **API/SDK version comparisons**: Alerts when documented versions fall behind latest releases.
91+
92+
### Contributor tooling
93+
94+
- **Pre-commit hooks**: Using [husky](https://github.com/typicode/husky) for:
95+
- Markdown linting.
96+
- Spell checking.
97+
- Format validation.
98+
- Link checking (local).
99+
- **Local validation scripts**: Easy-to-run scripts for common validation tasks.
100+
- **Documentation**: Guide for setting up and using local tooling.
101+
102+
### Freshness tracking
103+
104+
- **Documentation freshness dashboard**: Visibility into documentation age across sections.
105+
- **Update priority scoring**: System to prioritize which content needs updates most urgently.
106+
- **Maintainer notifications**: Automated notifications for high-priority updates.
107+
108+
## Timeline
109+
110+
- **Month 1**: Audit current automation, document requirements.
111+
- **Month 2**: Implement pre-commit hooks and local tooling.
112+
- **Month 3**: Fix version update automation for patch releases.
113+
- **Month 4**: Implement mdatagen integration for Collector metrics.
114+
- **Month 5**: Build content freshness tracking system.
115+
- **Month 6**: Testing, documentation, and rollout.
116+
117+
## Labels
118+
119+
- `docs:CI/infra`
120+
- `sig:comms`
121+
- `sig:collector`
122+
123+
## Related issues
124+
125+
- [#4523 - Automate the Collector list of internal metrics using `mdatagen` docs](https://github.com/open-telemetry/opentelemetry.io/issues/4523) (open) - mdatagen integration for metrics documentation.
126+
- [#7546 - Collector patch release updates break build because not all components are included](https://github.com/open-telemetry/opentelemetry.io/issues/7546) (open) - Version automation issues.
127+
- [#8313 - Set up git pre-commit and other hooks to make for a smoother contributor experience](https://github.com/open-telemetry/opentelemetry.io/issues/8313) (open) - Pre-commit hooks using husky.
128+
129+
## Project Board
130+
131+
To be created upon project approval.
132+
133+
## SIG Meetings and Other Info
134+
135+
This project will be coordinated through SIG Communications meetings with input from SIG Collector.
136+
137+
- **Slack channel**: `#otel-comms`
138+
- **Meeting notes**: To be linked upon project start.

0 commit comments

Comments
 (0)