Exp bdd tests (#6965)

* experimental scenario creation

* experimental scenario creation

* wip: structure and test execution

* wip: dependencies

* wip: changed test scenarios wording

* wip: k8s apply

* improve init of tests

* fix bug on state

* model embeds world to be able to perform world type operations

* changed context and model labels

* client with watcher

* wip: storage watcher

* fix scenario bug

* docs

* expose k8s client

* feat: watcher storage, saves the latest state of crds to later assert on state

* feat: watcher with wait for condition interface

* docs: added todos and commented feature test

Author: MiguelAAe

tests/integration/godog/architecture.md

@@ -0,0 +1,40 @@
+# Seldon Core 2 – Godog Test Architecture
+
+This document describes the architecture of the BDD-style test suites for Seldon Core 2
+using [godog](https://github.com/cucumber/godog) and Kubernetes.
+
+The goals of this architecture are:
+
+- Run **the same logical tests** against **different server configurations**.
+- Drive tests via **feature tags + config**, not hard-coded setup.
+- Provide a clean **domain-focused step API** (e.g. “models”, “pipelines”, …).
+- Maintain an **in-memory view of Kubernetes resources** to make assertions easy and fast.
+- Have flexibility to add future dependencies such as k6 or chaosmonkey
+
+---
+
+## 1. High-Level Overview
+
+At a high level:
+
+- **`TestMain`** creates and runs a `godog.TestSuite`.
+- **`InitializeTestSuite`** creates long-lived test dependencies:
+    - Kubernetes client(s)
+    - A Kubernetes watcher for CRDs with `test-suite=godogs`
+    - Reads/configures server setup from flags/config
+    - Optionally deploys server replicas and other shared infra
+- **`InitializeScenario`** runs per scenario:
+    - Creates a fresh **World** object (per-scenario state holder)
+    - Resets CRDs in the cluster (e.g. deletes test models)
+    - Creates a fresh **Model** for the scenario
+    - Registers domain-specific steps (e.g. model steps) against this World/Model
+- **Feature files** (Gherkin) describe behavior in domain language:
+    - e.g. “Given I have an "iris" model … Then the model should eventually become Ready”
+- **A watcher** keeps an up-to-date in-memory store of CRDs with label `test-suite=godogs` for fast, poll-free
+  assertions.
+
+## Run a test case
+
+```shell
+  go test --godog.tags='@0' --godog.concurrency=1 -race
+```

tests/integration/godog/features/autoscaling/model_autoscaling.feature

@@ -0,0 +1,2 @@
+#@AutoscalingModel
+#Feature: Model Autoscaling

tests/integration/godog/features/model/deployment.feature

@@ -0,0 +1,55 @@
+@ModelDeployment @Functional @Models
+Feature: Model deployment
+  In order to make a model available for inference
+  As a model user
+  I need to create a Model resource and verify it is deployed
+
+  @0
+  Scenario: Success - Load a model
+    Given I have an "iris" model
+    When the model is applied
+    Then the model should eventually become Ready
+
+
+  @0
+  Scenario: Success - Load a model again
+    Given I have an "iris" model
+    When the model is applied
+    Then the model should eventually become Ready
+
+#    this approach might be more reusable specially for complex test cases, its all how expressive we want to be
+  Scenario: Load model
+    Given I have a model:
+    """
+
+    """
+    When the model is applied
+    Then the model should eventually become Ready
+
+  Scenario: Success - Load a model and expect status model available
+    Given I have an "iris" model
+    When the model is applied
+    And the model eventually becomes Ready
+    Then the model status message should eventually be "ModelAvailable"
+
+  Scenario: Success - Load a model with min replicas
+    Given I have an "iris" model
+    And the model has "1" min replicas
+    When the model is applied
+    Then the model should eventually become Ready
+
+
+  Scenario: Success - Load a big model
+    Given I have an "large-model" model
+    When the model is applied
+    Then the model should eventually become Ready
+
+#    this would belong more to the feature of model server scheduling or capabilities
+  Scenario: Fail Load Model - no server capabilities in cluster
+    Given Given I have an "iris" model
+    And the model has "xgboost" capabilities
+    And there is no server in the cluster with capabilities "xgboost"
+    When the model is applied
+    Then the model eventually becomes not Ready
+    And the model status message should eventually be "ModelFailed"
+

tests/integration/godog/features/model/disruption.feature

@@ -0,0 +1,13 @@
+#@ModelDisruption @Models @Disruption
+#Feature: Model resilence to Core 2 disruption
+#
+#  Background:
+#    Given a clean test namespace
+#    And a Ready model "resilient-model" with capabilities "mlserver"
+#
+#  Scenario: Model keeps serving during a control plane restart
+#    Given a load test of 100 RPS is running against model "resilient-model"
+#    When I restart the Seldon Core 2 control plane
+#    Then at least 99% of requests should succeed during "2m"
+#    And the 95th percentile latency during "2m" should be less than "250ms"
+#    And no outage should last longer than "10s"

tests/integration/godog/features/model/inference.feature

@@ -0,0 +1,16 @@
+#@ModelInference @Models @Inference
+#Feature Basic model inferencing
+#
+#  Background:
+#    Given a clean test namespace
+#
+#  Scenario: Model can serve prediction
+#    Given I have an "iris" model
+#    And the model is applied
+#    And the model eventually becomes Ready
+#    When I send a prediction request with payload:
+#      """
+#      { "inputs": [1.0, 2.0, 3.0] }
+#      """
+#    Then the response status should be 200
+#    And the response body should contain "predictions"

tests/integration/godog/features/model/load.feature

@@ -0,0 +1,12 @@
+#@ModelLoad @Models @Load
+#Feature: Model performance under load
+#
+##  the model replicas could default to the max replicas of the available ml server
+#  Background:
+#    Given a clean test namespace
+#    And a Ready model "load-model" with capabilities "mlserver"
+#
+#  Scenario: Model meets latency SLO at 200 RPS
+#    When I run a load test of 200 RPS for "2m" against model "load-model"
+#    Then the 95th percentile latency should be less than "150ms"
+#    And the error rate should be less than "1%"

tests/integration/godog/features/model/parametized.feature

@@ -0,0 +1 @@
+#@ModelParametrizedParameters

tests/integration/godog/features/model/partial_scheduling.feature

@@ -0,0 +1,13 @@
+#@ModelPartialScheduling
+#Feature: Partial Model Scheduling
+#  In order to make a model partially available for inference
+#  As a model user
+#  I need to create a Model resource with min replicas less than the available server replicas and replicas bigger than the available server replicas and verify that the model deploys the minimum amount of possible model replicas and becomes ready
+#
+#  Scenario: Success - Load a model with partial replicas
+#    Given I have an "iris" model
+#    And the model has "1" min replicas
+#    And the model has "5" max replicas
+#    And the model has "1" replicas
+#    When the model is applied
+#    Then the model should eventually becomes Ready

tests/integration/godog/features/model/storage_secrets.feature

@@ -0,0 +1 @@
+#@ModelStorageSecrets