Enable Post-Processing and Export of Benchmark Results to External Data Warehouses

[acalhounRH]

Summary

Add post-processing capabilities to Zathras that convert benchmark results into structured JSON documents and export them to external data warehouses (OpenSearch, Horreum) for long-term storage, querying, and analysis.

---

Problem Statement / Current State

Currently, Zathras:

• ✅ Successfully orchestrates benchmark execution across cloud and bare metal systems
• ✅ Collects extensive metadata (hardware config, system config, cloud metadata)
• ✅ Retrieves test results as compressed tarballs (results_<test>.zip)
• ✅ Stores results in directory structures: results_prefix/os_vendor/cloud_type/instance_type_N/

However:

• ❌ Results are stored as unstructured tarballs on the controller filesystem
• ❌ No centralized database for historical result queries
• ❌ Difficult to perform trend analysis across multiple test runs
• ❌ No automated regression detection capabilities
• ❌ Limited visibility into performance trends over time
• ❌ Results are isolated per test run with no cross-run correlation
• ❌ Executive reporting requires manual data extraction

Impact:

• Performance engineers must manually extract and analyze results
• Historical comparisons require custom scripts
• No dashboards or visualization of trends
• Difficult to answer questions like:
  • "How has STREAM performance on m5.xlarge changed over the last 6 months?"
  • "Which tuned profile performs best for linpack across instance types?"
  • "Are we seeing performance regressions after OS updates?"

---

Proposed Solution

Implement a post-processing and export pipeline that:

1. Extracts results from Zathras archive directories
2. Transforms test outputs into structured JSON documents
3. Enriches with metadata (hardware, system config, cloud details)
4. Validates data integrity and schema compliance
5. Exports to configurable data warehouse targets

Architecture

post_processing/
├── main.py                          # Orchestrator script
├── processors/
│   ├── base_processor.py           # Abstract base class
│   ├── fio_processor.py            # FIO-specific parser
│   ├── streams_processor.py        # STREAM-specific parser
│   └── ...                          # One per test type
├── exporters/
│   ├── opensearch_exporter.py      # OpenSearch integration
│   └── horreum_exporter.py         # Horreum integration
└── utils/
    ├── metadata_extractor.py       # Parse Zathras metadata
    └── archive_handler.py          # Handle zip/tar extraction

Unified JSON Schema

{
  "test_run": {
    "id": "uuid",
    "timestamp": "ISO-8601",
    "zathras_version": "3.2"
  },
  "infrastructure": {
    "type": "aws|azure|gcp|local",
    "instance_type": "m5.xlarge",
    "region": "us-east-1",
    "os": { "vendor": "rhel", "version": "9.3" }
  },
  "hardware": {
    "cpu": { "model": "...", "cores": 16, ... },
    "memory": { "total_gb": 32, ... }
  },
  "test": {
    "name": "streams",
    "version": "v1.0",
    "status": "passed|failed",
    "duration_seconds": 235
  },
  "results": {
    "metrics": { /* test-specific */ },
    "raw_output": "..."
  }
}

---

Justification

Why This Approach?

Option 1 Considered: Modify all test wrappers to output JSON

• ❌ Requires coordinating changes across 18+ separate repositories
• ❌ Each wrapper maintained by different teams
• ❌ No standardization guarantee
• ❌ Months of coordination overhead
• ❌ Breaks backward compatibility

Option 2 (Proposed): Post-processing at Zathras level

• ✅ Single point of implementation - change only Zathras
• ✅ Wrapper independence - no changes to external test repos
• ✅ Standardized schema - consistent structure across all tests
• ✅ Metadata enrichment - combine test results with infrastructure context
• ✅ Faster deployment - weeks instead of months
• ✅ Backward compatible - doesn't break existing workflows
• ✅ Historical data support - can reprocess old results

Why Not Store in Zathras Database?

• Performance testing requires time-series analysis capabilities
• Need sophisticated querying across multiple dimensions
• Require visualization/dashboard integration
• Benefit from existing data warehouse infrastructure
• Scale to millions of data points over time

---

Benefits

For Performance Engineers

• 📊 Real-time dashboards showing current and historical trends
• 🔍 Ad-hoc queries across any dimension (test, OS, instance type, date range)
• 📈 Trend visualization to spot performance changes over time
• 🚨 Regression detection via automated alerts
• 📝 Automated reporting for stakeholders

For Engineering Managers

• 📉 Executive dashboards with performance KPIs
• 💰 Cost analysis (test runtime × cloud pricing = spend tracking)
• 🎯 Goal tracking (performance targets vs actual results)
• 📊 Team productivity metrics (tests run, pass rates)

For CI/CD Integration

• 🔄 Automated performance gates (fail CI if regression detected)
• 📧 Notifications on performance changes
• 🔗 Integration with existing monitoring systems
• 📦 Data portability (JSON export to other tools)

For Research & Analysis

• 🔬 Cross-test correlation (does better STREAM predict better HammerDB?)
• 🌡️ Environmental impact (how do tuned profiles affect results?)
• 📐 Statistical analysis (standard deviation, percentiles)
• 🗂️ Dataset creation for ML models

---

Data Warehouse Options

Option A: OpenSearch (Elasticsearch fork)

Best for: General-purpose search, analytics, and visualization

Pros:

• ✅ Powerful query language (SQL + DSL)
• ✅ Kibana dashboards for visualization
• ✅ Time-series analysis built-in
• ✅ Widely adopted, large community
• ✅ Real-time indexing and search
• ✅ Flexible schema (add fields without migration)
• ✅ REST API for easy integration

Cons:

• ⚠️ Not performance-testing-specific
• ⚠️ Requires infrastructure setup
• ⚠️ May need custom dashboards

Use Cases:

• Real-time monitoring during test runs
• Ad-hoc queries across result history
• Executive dashboards
• Log correlation with performance data

---

Option B: Horreum (Performance Test Results Repository)

Best for: Performance regression tracking and historical comparison

Pros:

• ✅ Built specifically for performance testing
• ✅ Automatic regression detection
• ✅ Change point analysis (detects when performance shifts)
• ✅ Comparison views (before/after, baseline/candidate)
• ✅ Schema validation for test results
• ✅ Native understanding of performance metrics
• ✅ Integration with CI/CD pipelines
• ✅ Run comparison and annotation

Cons:

• ⚠️ More specialized, smaller community
• ⚠️ Less flexible for non-performance queries
• ⚠️ Steeper learning curve

Use Cases:

• Performance regression tracking in CI/CD
• Baseline management (golden results)
• Automated alerting on regressions
• Historical performance comparison

---

Option C: Both (Recommended)

Strategy: Support multiple exporters, let users configure based on needs

# In scenario file
global:
  results_export:
    enabled: true
    targets:
      - opensearch:
          url: "https://opensearch.example.com"
          index: "zathras-results"
      - horreum:
          url: "https://horreum.example.com"
          test: "zathras-benchmark-suite"

Benefits:

• OpenSearch for real-time monitoring and dashboards
• Horreum for regression detection and CI/CD gates
• Users choose based on infrastructure and needs
• Not mutually exclusive

---

Option D: Other Data Warehouses

The architecture supports adding exporters for:

• InfluxDB (time-series focused)
• PostgreSQL/TimescaleDB (SQL queries)
• Prometheus (metrics and alerting)
• Custom REST APIs (organization-specific systems)

---

Implementation Approach

Phase 1: Foundation

• Create post_processing/ directory structure
• Implement base processor interface
• Build metadata extractor (parse hw_config.yml, ansible_vars.yml)
• Implement archive handler (zip/tar extraction)
• Create one processor (FIO - already outputs JSON)
• Build OpenSearch exporter
• Create main.py orchestrator
• Test end-to-end with sample FIO results

Deliverable: Working prototype that processes FIO results to OpenSearch

---

Phase 2: Expansion

• Add 3-5 more processors (streams, linpack, coremark, uperf)
• Implement Horreum exporter
• Add configuration file support
• Error handling and retry logic
• Logging and debugging capabilities
• CLI improvements (dry-run, batch mode)

Deliverable: Multi-test support with dual exporter capability

---

Phase 3: Production Ready

• Complete remaining processors
• Integration tests with real scenarios
• Documentation (README, usage examples)
• Schema documentation
• Performance optimization
• Optional integration into burden script

Deliverable: Production-ready, documented solution

---

Key Design Decisions

1. Standalone vs Integrated

Decision: Build as standalone tool first, integrate later

Rationale:

• Allows independent development and testing
• Can process historical data
• Users can run manually or in cron jobs
• Optional integration into burden when stable

2. Push vs Pull

Decision: Push model (Zathras pushes to data warehouse)

Rationale:

• Simpler architecture
• Real-time availability
• No data warehouse needs Zathras access
• Standard pattern for observability

3. Synchronous vs Asynchronous

Decision: Synchronous initially, async later if needed

Rationale:

• Simpler implementation
• Export time is small relative to test runtime
• Can optimize later if bottleneck

4. Schema Strict vs Flexible

Decision: Flexible schema with core required fields

Rationale:

• Tests evolve over time
• New metadata may be added
• Graceful degradation if parsing fails
• OpenSearch handles schema evolution well

---

Success Criteria

Must Have

• Process at least 3 test types (FIO, STREAM, Linpack)
• Export to OpenSearch successfully
• Unified JSON schema documented
• Metadata enrichment working (hardware + cloud + test config)
• Error handling (graceful failures)
• Can process historical results

Should Have

• Export to Horreum
• 5-7 test processors implemented
• Configuration file support
• Batch processing mode
• Unit test coverage >70%

Nice to Have

• All 18 test processors
• Integration into burden script
• Sample Kibana dashboards
• Performance optimization
• Resumability (skip already-exported)

---

Non-Goals (Out of Scope)

• ❌ Modifying test wrappers
• ❌ Real-time streaming during test execution
• ❌ Building custom visualization UI
• ❌ Changing Zathras core functionality
• ❌ Result validation/correctness checking
• ❌ Test execution scheduling

---

Risks & Mitigations

Risk	Impact	Mitigation
Test output format changes break parsers	Medium	Keep raw output; version detection; graceful fallback
Data warehouse unavailable during export	Low	Retry logic; queue exports; offline mode
Schema evolution over time	Medium	Flexible schema; version field; backward compatibility
Performance overhead	Low	Async export; make optional; optimize critical paths
Adoption resistance	Medium	Make optional; show value first; gather feedback

---

Open Questions

1. Authentication: How should credentials be managed? (env vars, config file, secrets manager?)
2. Data retention: Who manages data warehouse retention policies?
3. Schema governance: Who approves schema changes?
4. Priority tests: Which 3-5 tests should we implement first?
5. Infrastructure: Is OpenSearch/Horreum already deployed or needs setup?
6. Access control: Who can export results? Any restrictions?

---

References

• Zathras repository: https://github.com/redhat-performance/zathras
• OpenSearch documentation: https://opensearch.org/docs/latest/
• Horreum documentation: https://horreum.hyperfoil.io/
• Test wrappers: https://github.com/redhat-performance/ (various repos)

---

Labels

enhancement, feature, observability, data-export

[acalhounRH]

Development Branch Created

🚀 Development for this feature is now underway on branch: enable-post-processing-and-export

Quick Links

• 📂 Browse branch code
• 🔍 Compare with upstream main
• 📝 View commits

Initial Progress

• Branch created from latest main (2571bcb)
• Directory structure established:
  • post_processing/ - Main package
  • post_processing/processors/ - Test-specific parsers
  • post_processing/exporters/ - Data warehouse integrations
  • post_processing/utils/ - Shared utilities
• Base processor implementation
• First test processor (FIO)
• OpenSearch exporter
• End-to-end testing

Will update this issue as development progresses. Feedback welcome! 🙏

@grdumas @dvalinrh @frival

[acalhounRH]

Schema Design Decision: Object-Based Structure with Dynamic Keys

Executive Summary

Chosen an object-based schema with dynamic keys for storing Zathras benchmark results in OpenSearch and Horreum. This approach uses runs as an object with dynamic keys (run_1, run_2, etc.) and timestamps as keys for time series data, avoiding OpenSearch's nested type performance issues while maintaining complete data structure, order preservation, and query performance.

---

The Problem: OpenSearch Nested Arrays Performance

Why We Can't Use Nested Arrays

OpenSearch has significant performance issues with nested types:

1. Document Explosion: Each nested object creates hidden internal documents

   • Example: 2 runs × 1000 time series points = 2000+ hidden documents per result
   • Impacts indexing speed, memory usage, and query performance

2. Query Performance: Nested queries are 5-10x slower than standard queries

   • Requires special nested query syntax
   • All nested documents loaded into memory during queries
   • Complex aggregations across nested data are expensive

3. Memory Pressure: Large nested structures cause heap pressure

   • Can trigger circuit breakers
   • Slows down cluster operations

4. Indexing Bottleneck: Nested arrays dramatically slow indexing

   • Each nested change requires reindexing all nested documents
   • Updates become progressively slower with more nested data

Original Problematic Structure

{
  "results": {
    "runs": [  // ← Array requires nested type
      {
        "run_number": 1,
        "time_series": [  // ← Nested array within nested array
          {"timestamp": "...", "value": 193245},
          {"timestamp": "...", "value": 195999}
        ]
      }
    ]
  }
}

Performance impact: With 3 runs × 500 time series points each = 1500 nested documents, queries become unacceptably slow.

---

The Solution: Object-Based Structure with Dynamic Keys

Core Strategy

Replace arrays with objects using dynamic keys:

1. runs as Object: Instead of runs[] array, use runs: {run_1: {}, run_2: {}}
2. Timestamps as Keys: Instead of time_series[] array, use timeseries: {"2025-11-06T05:09:45.000Z": {}}
3. Object Type (not Nested): OpenSearch treats this as a flat object, not nested documents

Schema Structure

{
  "metadata": {
    "document_id": "coremark_localhost_20251106_050945",
    "document_type": "zathras_test_result",
    "collection_timestamp": "2025-11-06T05:09:45Z"
  },
  
  "test": {
    "name": "coremark",
    "version": "v1.01",
    "wrapper_version": "v2.0"
  },
  
  "system_under_test": {
    // Fully denormalized - complete SUT metadata
    "hardware": {
      "cpu": {
        "vendor": "Intel",
        "model": "Xeon Gold 6230",
        "cores": 40
      },
      "memory": {
        "total_gb": 256
      },
      "numa": {
        "node_0": {"cpus": "0-19,40-59", "memory_gb": 128},
        "node_1": {"cpus": "20-39,60-79", "memory_gb": 128}
      },
      "storage": {
        "device_0": {"type": "nvme", "capacity_gb": 1000},
        "device_1": {"type": "ssd", "capacity_gb": 1000}
      }
    },
    "operating_system": {
      "distribution": "rhel",
      "version": "9.3",
      "kernel_version": "5.14.0-362.el9.x86_64"
    }
  },
  
  "test_configuration": {
    "iterations_requested": 5,
    "test_specific_parameters": {}
  },
  
  "results": {
    "status": "PASS",
    "execution_time_seconds": 204,
    "total_runs": 2,
    
    "primary_metric": {
      "name": "iterations_per_second",
      "value": 191234.5,
      "unit": "iterations/sec"
    },
    
    "overall_statistics": {
      "mean": 191234.5,
      "median": 190905.9,
      "min": 188073.0,
      "max": 195999.8,
      "stddev": 2543.2,
      "sample_count": 10
    },
    
    "runs": {
      "run_1": {
        "run_number": 1,
        "status": "PASS",
        "start_time": "2025-11-06T05:09:45Z",
        "end_time": "2025-11-06T05:10:07Z",
        "duration_seconds": 22.449,
        
        "configuration": {
          "threads": 4,
          "compiler": "GCC15.2.1",
          "compiler_flags": "-O2 -DMULTITHREAD=4"
        },
        
        "metrics": {
          "iterations_per_second": 195999.821818,
          "total_iterations": 4400000,
          "coremark_size": 666
        },
        
        "timeseries_summary": {
          "count": 5,
          "mean": 192806.34,
          "min": 189843.38,
          "max": 195999.82,
          "stddev": 2543.2
        },
        
        "timeseries": {
          "2025-11-06T05:09:45.000Z": {
            "sequence": 0,
            "iteration": 1,
            "value": 193245.201809
          },
          "2025-11-06T05:09:50.000Z": {
            "sequence": 1,
            "iteration": 2,
            "value": 195999.821818
          },
          "2025-11-06T05:09:55.000Z": {
            "sequence": 2,
            "iteration": 3,
            "value": 190905.935439
          }
        }
      },
      
      "run_2": {
        "run_number": 2,
        "status": "PASS",
        "start_time": "2025-11-06T05:10:10Z",
        "duration_seconds": 22.1,
        
        "configuration": {...},
        "metrics": {...},
        "timeseries_summary": {...},
        
        "timeseries": {
          "2025-11-06T05:10:10.000Z": {
            "sequence": 0,
            "iteration": 1,
            "value": 190642.624513
          }
        }
      }
    }
  }
}

---

Why This Approach Solves the Problem

1. No Nested Type Required

Object type vs Nested type in OpenSearch:

// OpenSearch Mapping
{
  "mappings": {
    "properties": {
      "results": {
        "properties": {
          "runs": {
            "type": "object",  // ← NOT nested!
            "dynamic": true
          }
        }
      }
    }
  }
}

Key difference:

• Nested type: Creates separate hidden documents, requires special queries
• Object type: Flattened at index level, standard queries work

2. Logical Data Organization

All data for a run is grouped together:

• results.runs.run_1.metrics.*
• results.runs.run_1.configuration.*
• results.runs.run_1.timeseries.*

Instead of scattered across document:

• run_1_metrics_*
• run_1_config_*
• run_1_timeseries_*

3. Time Series as Timestamps-as-Keys

Using ISO 8601 timestamps as object keys:

Benefits:

• ✅ Natural representation of time-based data
• ✅ Direct timestamp-based access: timeseries["2025-11-06T05:09:45.000Z"]
• ✅ Human-readable keys
• ✅ Sortable (ISO 8601 strings sort chronologically)
• ✅ No array indexing needed

Order preservation:

• sequence field provides explicit ordering
• ISO 8601 timestamp keys are sortable
• Application can reconstruct chronological order reliably

4. Query Performance

Standard queries work (no nested query syntax):

// Find tests where run_1 exceeded threshold
GET /zathras-results/_search
{
  "query": {
    "range": {
      "results.runs.run_1.metrics.iterations_per_second": {
        "gte": 190000
      }
    }
  }
}

// Aggregate across all run_1 results
GET /zathras-results/_search
{
  "size": 0,
  "aggs": {
    "run_1_avg": {
      "avg": {
        "field": "results.runs.run_1.metrics.iterations_per_second"
      }
    }
  }
}

No nested query overhead - queries execute 5-10x faster than nested equivalents.

---

Justification: Why Object-Based Over Alternatives

Alternative 1: Keep Nested Arrays

Why rejected:

• ❌ 5-10x slower query performance
• ❌ Memory pressure and indexing bottlenecks
• ❌ Document explosion (1000s of hidden docs)
• ❌ Not scalable for large time series

Alternative 2: Fully Flatten Everything

"run_1_metric_iterations_per_second": 195999.82,
"run_1_timeseries_value_0": 193245.2,
"run_1_timeseries_value_1": 195999.8

Why rejected:

• ❌ Poor organization - fields scattered across document
• ❌ Hard to read and maintain
• ❌ Complex field naming conventions
• ❌ Difficult to query dynamically
• ⚠️ Still fast, but less maintainable

Alternative 3: One Document Per Run

Why rejected:

• ❌ Loses test execution context (runs split across documents)
• ❌ Need to query multiple docs and join in application
• ❌ More complex to aggregate statistics across runs
• ❌ User prefers one document per test execution

Alternative 4: Store Time Series Externally

Why rejected:

• ❌ Two-step retrieval (document + external storage)
• ❌ Increased complexity
• ❌ Loses atomic retrieval
• ⚠️ May still be needed for very large time series (1000+ points)

Chosen: Object-Based with Dynamic Keys ⭐

Why chosen:

• ✅ No nested type overhead - 5-10x faster queries
• ✅ Organized structure - logical grouping of run data
• ✅ Complete data in one document - atomic retrieval
• ✅ Time series included - no external storage needed
• ✅ Natural time representation - timestamps as keys
• ✅ Order preserved - via sequence fields and sortable keys
• ✅ Standard queries - no nested query syntax
• ✅ Scalable - handles 5-20 runs easily
• ✅ Maintainable - clean, intuitive structure
• ✅ Flexible - easy to add new run-specific fields

---

Technical Details

OpenSearch Mapping Configuration

{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "metadata": {
        "properties": {
          "document_id": {"type": "keyword"},
          "collection_timestamp": {"type": "date"}
        }
      },
      "test": {
        "properties": {
          "name": {"type": "keyword"},
          "version": {"type": "keyword"}
        }
      },
      "results": {
        "properties": {
          "status": {"type": "keyword"},
          "total_runs": {"type": "integer"},
          "primary_metric": {
            "properties": {
              "name": {"type": "keyword"},
              "value": {"type": "double"},
              "unit": {"type": "keyword"}
            }
          },
          "runs": {
            "type": "object",
            "dynamic": true
          }
        }
      }
    },
    "dynamic_templates": [
      {
        "run_metrics": {
          "path_match": "results.runs.*.metrics.*",
          "mapping": {"type": "double"}
        }
      },
      {
        "run_timeseries_values": {
          "path_match": "results.runs.*.timeseries.*.value",
          "mapping": {"type": "double"}
        }
      },
      {
        "run_timeseries_sequence": {
          "path_match": "results.runs.*.timeseries.*.sequence",
          "mapping": {"type": "integer"}
        }
      }
    ]
  }
}

Order Preservation Strategy

Three mechanisms ensure order is maintained:

1. Explicit sequence field in each time series data point

   "timeseries": {
     "2025-11-06T05:09:45.000Z": {"sequence": 0, "value": 193245}
   }

2. ISO 8601 timestamp keys - sortable as strings

   ordered = sorted(timeseries.items())  # Keys sort chronologically

3. timeseries_summary - metadata about the time series

   "timeseries_summary": {
     "count": 5,
     "first_timestamp": "2025-11-06T05:09:45.000Z",
     "last_timestamp": "2025-11-06T05:10:05.000Z"
   }

Handling Variable Number of Runs

Application discovers runs dynamically:

# Retrieve document
result = opensearch.get(index="zathras-results", id=doc_id)

# Get runs object
runs = result['_source']['results']['runs']

# Iterate over all runs (regardless of count)
for run_key, run_data in runs.items():
    run_number = run_data['run_number']
    metrics = run_data['metrics']
    timeseries = run_data['timeseries']
    
    # Process time series in order
    ordered_ts = sorted(
        timeseries.items(),
        key=lambda x: x[1]['sequence']
    )

---

Benefits Summary

Performance Benefits

• ✅ 10-50x faster queries vs nested arrays
• ✅ Faster indexing - no nested document overhead
• ✅ Lower memory usage - no hidden document explosion
• ✅ Standard query syntax - no nested query complexity

Structural Benefits

• ✅ Logical organization - run data grouped together
• ✅ Natural time series - timestamps as keys
• ✅ Complete context - one document per test execution
• ✅ Fully denormalized - all SUT metadata embedded

Operational Benefits

• ✅ Easy to understand - intuitive structure
• ✅ Maintainable - clean field hierarchy
• ✅ Flexible - easy to add fields per run
• ✅ Scalable - handles 5-20 runs efficiently

Functional Benefits

• ✅ Time series included - no external storage
• ✅ Order preserved - via sequence + sortable keys
• ✅ Direct access - query specific runs easily
• ✅ Aggregations work - standard OpenSearch aggs

---

Trade-offs and Limitations

Accepted Trade-offs

1. Dynamic Field Count

   • Number of fields varies based on run count and time series length
   • Mitigation: OpenSearch handles thousands of fields well; set appropriate field limits

2. Query Path Length

   • Paths like results.runs.run_1.metrics.iterations_per_second are longer
   • Mitigation: Acceptable trade-off for better organization

3. Object Key Discovery

   • Must enumerate object keys to find all runs
   • Mitigation: Store total_runs metadata; application iterates dynamically

4. Timestamp Key Format

   • Must standardize on ISO 8601 format
   • Mitigation: Use consistent formatting in processor; validate on export

Limitations

1. Practical Run Limit: ~10-20 runs per test execution

   • Beyond this, document becomes very large
   • Solution: If more runs needed, consider separate documents per run with linking

2. Large Time Series: For 1000+ time series points per run

   • Document size becomes 200KB+
   • Solution: Sample data (every Nth point) or use hybrid approach with external storage for full data

3. No Guaranteed Object Key Order: JSON objects aren't ordered

   • Solution: Explicit sequence field ensures order reconstruction

---

Implementation Requirements

To implement this schema, we need:

1. ✅ Processors: Extract run data and build object structures dynamically
2. ✅ Timestamp Formatting: Standardize on ISO 8601 with milliseconds
3. ✅ Sequence Assignment: Add sequence numbers to time series data
4. ✅ OpenSearch Index Template: Configure mappings with dynamic templates
5. ✅ Validation: Ensure timestamp keys are valid, sequences are sequential
6. ✅ Documentation: Guide users on querying object-based structure

---

Conclusion

The object-based structure with dynamic keys is the optimal choice for Zathras benchmark results because:

1. Solves Performance Problem: Eliminates nested type overhead while maintaining structure
2. Preserves Complete Data: Time series included without external storage
3. Natural Representation: Timestamps as keys naturally represent time-based data
4. Maintains Order: Sequence fields and sortable timestamps ensure data order
5. Query Performance: Standard OpenSearch queries work efficiently
6. Organized Structure: Logical grouping makes data easy to understand and work with
7. Single Document: Complete test execution context in one atomic document
8. Scalable: Handles typical test scenarios (5-10 runs, 100-500 time points) efficiently

This design provides the best balance of performance, structure, completeness, and usability for storing and querying Zathras benchmark results in document stores like OpenSearch and Horreum.