Context-Aware XBRL Standardization: Community Feedback

[dgunning]

Background: Issue #494 requested documentation on XBRL standardization customization. While completing that request, we conducted comprehensive research comparing EdgarTools' approach with @mpreiss9's methodology for handling 200+ companies with ambiguous XBRL tags.

📚 What's Been Delivered

✅ Comprehensive Documentation (Issue #494 - Complete)

• Guide: docs/advanced/customizing-standardization.md (2,408 lines)
• Covers current system, limitations, customization patterns
• Includes real-world examples (Tesla, Microsoft, Berkshire Hathaway)

✅ Research & Analysis (Complete)

• Detailed comparison: EdgarTools vs @mpreiss9's approach
• Identified 7 specific enhancements that could improve EdgarTools
• Research document: docs-internal/research/issues/issue-494-standardization-comparison.md

✅ Enhancement Roadmap (Planned)

• 5 implementation phases documented
• Effort estimates: 8-12 weeks across multiple releases
• Roadmap: docs-internal/planning/future-enhancements/context-aware-standardization.md

🎯 Key Finding

The two approaches are complementary, not contradictory:

• EdgarTools: Simple, fast for 95% of non-ambiguous tags
• @mpreiss9's method: Robust context-aware resolution for 200+ ambiguous tags
• Hybrid approach: Best of both worlds

🚀 Proposed Enhancements

Phase 1-2: Foundation (v4.30.0) - Ready to Start

Phase 1: Balance Sheet Validation (1-2 weeks)

• Validate: Assets = Liabilities + Equity
• Opt-in feature, no breaking changes
• Benefits ALL users (catches data quality issues)

Phase 2: Section Membership Dictionary (1 week)

• Define which concepts belong in which sections
• Foundation for context-aware resolution
• Pure data definition, low risk

Phase 3-5: Advanced Features (v4.31.0+) - Needs Feedback

Phase 3: Enhanced Context Threading (2-3 weeks)

• Thread calculation parent through standardization
• Enable section-based disambiguation
• Medium complexity

Phase 4: Context-Aware Disambiguation (3-4 weeks)

• Handle 200+ ambiguous XBRL tags
• Section-based resolution using @mpreiss9's methodology
• Core enhancement, medium-high risk

Phase 5: Unmapped Tag Logging (1-2 weeks)

• CSV-based workflow for continuous improvement
• Excel-friendly for managing large mapping sets
• Suggested mappings with confidence scores

💬 Community Feedback Needed

We'd like YOUR input on priorities before implementing Phases 3-5:

Question 1: What's your highest priority?

A. Balance Sheet Validation (Phase 1)

• Catch data quality issues early
• Validate accounting equations
• Production-ready validation

B. Context-Aware Disambiguation (Phase 4)

• Handle ambiguous XBRL tags correctly
• Section-based resolution
• Solve the 200+ ambiguous tag problem

C. CSV Workflow Enhancements (Phase 5)

• Excel-friendly mapping management
• Unmapped tag discovery
• Continuous improvement workflow

D. All of the above (Full 5-phase implementation)

Question 2: What's your use case?

• Managing 1-10 companies?
• Managing 10-100 companies?
• Managing 200+ companies like @mpreiss9?
• Academic research?
• Production financial applications?

Question 3: What's your pain point?

• Data quality issues (validation would help)?
• Ambiguous tags mapping incorrectly?
• Managing large mapping sets?
• Something else?

🗓️ Timeline

v4.29.0 (Week of 2025-11-25): Documentation complete
v4.30.0 (Q1 2025): Phase 1-2 (if community feedback positive)
v4.31.0+ (TBD): Phase 3-5 (conditional on feedback)

📖 Resources

• Closed Issue: Create documentation on how to customize standardization tagging #494 (documentation request - complete)
• Research Document: docs-internal/research/issues/issue-494-standardization-comparison.md
• Enhancement Roadmap: docs-internal/planning/future-enhancements/context-aware-standardization.md
• User Guide: docs/advanced/customizing-standardization.md

🙏 Special Thanks

@mpreiss9 - Your detailed methodology for handling 200+ companies with CSV workflows and section-based resolution was invaluable. The research shows your approach works beautifully alongside EdgarTools' current system. We'd love your continued input as we consider these enhancements!

---

Please share your priorities, use cases, and feedback below! 👇

This discussion will inform whether we proceed with Phases 3-5 and help us prioritize features that provide the most value to the community.

[mpreiss9]

Not sure if you saw, I added additional information to #494 after it was closed.

[dgunning]

I converted Issue 494 to discussion 502. We can continue there