Benchmark structure setup

[Martin187187]

User Story: Benchmark Setup for BaSyx Endpoints

User Story

As a BaSyx developer,
I want a standardized benchmark setup for BaSyx endpoints,
so that scalability, performance, and runtime behavior can be
verified in a reproducible way.

---

Scope and Goals

The benchmark setup is intended to:

• Measure runtime behavior of read and write endpoints under increasing data volume
• Support both quick performance checks and full end-to-end scalability tests
• Be easily extensible to additional BaSyx components without code changes

---

Benchmark Modes

1. Quick Runtime Test (Warm System)

• Uses an already deployed BaSyx setup
• Database is pre-filled with test data
• Focuses on runtime performance of GET and other read-heavy endpoints
• Optimized for fast feedback during development

2. Full Build-Up Test (Cold System)

• Starts from a clean deployment (empty database)
• Automatically populates the database with benchmark data
• Executes the full set of endpoints (CRUD+)
• Measures both build-up time and steady-state runtime behavior

---

Rules and Constraints

• A fully deployed setup is required, including BaSyx components and a
  database
• For every write endpoint, a template definition must exist
  specifying:
  • Data structure (payload schema)
  • Execution frequency
  • Expected data volume growth
• Benchmark configuration must be data-driven (JSON-based)

---

Entry Points

The initial benchmark scope includes: - AAS Registry - Basic Discovery
Service - Digital Twin Registry (DTR)

---

Acceptance Criteria

This task is considered complete when:

• Complex benchmark runs support full system build-up from an empty database
• Quick runtime tests run on already deployed and populated systems
• New components/endpoints can be added via JSON configuration only
• Runtime plots show stable query behavior as data volume increases
• Benchmarks successfully run for AAS Registry, Basic Discovery, and DTR

---

Open Issue: Schema Changes in Warm Runtime Mode

Problem

Warm-mode benchmarks assume a compatible, populated database. Schema
changes introduced by new pull requests may break this assumption.

Handling Strategy

• Warm-mode benchmarks only run if schema compatibility is guaranteed
• Backward-compatible changes should use non-breaking migrations
  (e.g. ALTER)
• If incompatible changes are detected:
  • Warm-mode is skipped
  • Benchmark falls back to Cold System mode automatically

Acceptance Criteria Extension

• Benchmark execution detects schema incompatibilities and selects
  the correct mode

---

Deliverables

• JSON-based benchmark configuration templates
• Scripts for warm and cold benchmark execution
• Runtime and scalability plots
• Documentation for extending benchmarks