[Planning] Metadata ingestion automation approaches

[tomvothecoder]

Document to host discussion: https://e3sm.atlassian.net/wiki/spaces/IPD/pages/5684461604/12-02-25+Strategies+for+Automating+E3SM+Simulation+Metadata+Ingestion+into+SimBoard

Summary

Explore automation approaches for ingesting E3SM simulation metadata into the Sim Viewer.

Details

This task focuses on evaluating multiple approaches for automating simulation metadata ingestion into the Sim Viewer (EarthFrame). The goal is to determine a sustainable, maintainable path forward given current architecture and resourcing constraints.

Objectives:

• Assess two near-term ingestion strategies:
  1. Extend PACE ingestion to feed Sim Viewer.
  2. Implement PACE parsing logic directly in Sim Viewer.
• Document pros/cons, refactor requirements, deployment implications, and FTE needs.
• Consider a long-term architectural path:
  • A unified metadata backend providing canonical parsing + metadata services for all tools.
• Produce a recommended short-, mid-, and long-term plan.

Stakeholders:

• SimBoard development team
• E3SM infrastructure team

Deliverables

• Summary of ingestion approach feasibility and tradeoffs.
• Recommended phased integration plan.
• Estimated effort and FTE requirements for each approach.
• Notes on potential unified backend direction.

Dependencies & Blockers

• Coordination with DOE HPC environments for ingestion testing.
• Limited FTE resources for parser refactoring.
• Need access to relevant case directories and existing PACE parser logic.

Due Date

No response