Feature/cli validation (#92)

* fix: remove unused flake8 im pre-commit

* refactor: cli validation in configure commands

* feat: move validation to pydantic

* feat: add pydantic alternatives validation

* refactor: drup unused

* fix: criteria check

* feat: add error handling

* fix: power report

* fix: alternative command

* feat: add validation for cv data dependency

* add tests

* feat: Add 3-step experiment workflow example

* fix: test

* fix: cli

Author: faux66

.pre-commit-config.yaml

@@ -1,12 +1,6 @@
 # See https://pre-commit.com for more information
 # See https://pre-commit.com/hooks.html for more hooks
 repos:
-  - repo: https://github.com/pycqa/flake8
-    rev: "7.1.1"
-    hooks:
-      - id: flake8
-        additional_dependencies: [Flake8-pyproject]
-        # stages: [push]
 
   - repo: https://github.com/pre-commit/mirrors-mypy
     rev: "v1.13.0"

experiment_example/README.md

@@ -0,0 +1,136 @@
+# Quick-Start Template: A 3-Step Experimental Workflow
+
+This directory provides a set of template configuration files for the `pysatl-experiment` framework. It is designed to serve as a starting point for conducting a robust, three-step comparison of statistical goodness-of-fit tests.
+
+The goal of this workflow is to help you make a data-driven decision about which statistical test is best suited for your specific hypothesis and potential alternatives.
+
+## The 3-Step Workflow
+
+This template follows a complete research cycle. Each step answers a different question, and they are designed to be run in sequence.
+
+---
+
+### Step 1: Calibration
+File: `1_calculate_critical_values.json`
+
+**The Question:** What are the correct decision thresholds for my chosen tests?
+
+**The Purpose:** Before you can use any statistical test, you must establish its baseline critical values. These values depend on your null hypothesis (the distribution you are testing for), the sample sizes, and the desired significance levels (alpha). This step calibrates your "measurement tools."
+
+**The Command:**
+```bash
+poetry run experiment build-and-run 1_calculate_critical_values
+```
+---
+### Step 2: Evaluation 
+File: `2_evaluate_power.json`
+
+**The Question:** How effective are these tests at detecting the specific deviations I care about?
+
+**The Purpose:** This is the core scientific evaluation. You measure the **statistical power** of each test against one or more alternative hypotheses. A test with high power is very good at correctly rejecting the null hypothesis when it is indeed false. This step tells you which test is most effective for your specific problem.
+
+*This experiment requires the critical values generated in Step 1.*
+
+**The Command:**
+```bash
+poetry run experiment build-and-run 2_evaluate_power
+```
+
+---
+
+### Step 3: Performance Analysis
+File: `3_measure_time_complexity.json`
+
+**The Question:** What is the computational cost of using each test?
+
+**The Purpose:** This is the practical, engineering evaluation. If two tests show similar power, the faster one may be preferable, especially in data-intensive applications. This experiment measures the execution time of each test, helping you understand the trade-offs between effectiveness and performance.
+
+**The Command:**
+```bash
+poetry run experiment build-and-run 3_measure_time_complexity
+```
+
+---
+
+## How to Adapt This Template for Your Own Research
+
+To use these files for your own experiment, copy this directory and modify the JSON configurations. The key is to ensure your parameters are consistent across the files.
+#### 1. Define Your Null Hypothesis
+
+In all three files (`1_...`, `2_...`, `3_...`), change the `hypothesis` object to match the distribution you want to test for.
+
+**Example: Changing from `Normal` to `Weibull`**
+```json
+  "hypothesis": "normal"
+to
+  "hypothesis": "weibull"
+```
+
+#### 2. Choose the Criteria to Compare
+
+In all three files, update the `criteria` list to include the tests you want to evaluate.
+
+**Example: Comparing `Liliefos` and `Chi-Squared`**
+```json
+    "criteria": [
+      {
+        "criterion_code": "LILLIE",
+        "parameters": []
+      },
+      {
+        "criterion_code": "CHI2",
+        "parameters": []
+      }
+    ],
+```
+#### 3. Define the Alternative Hypothesis
+
+In `2_evaluate_power.json`, modify the `alternatives` array. This is the specific deviation you want your tests to be able to detect.
+
+**Example: Checking if tests for `Weibull` can detect an `Exponential` distribution**
+```json
+"alternatives": [
+      {
+        "generator_name": "EXPONENTIALGENERATOR",
+        "parameters": [
+          0.5,
+        ]
+      }
+    ]
+```
+#### 4. Adjust Experiment Parameters
+
+You can also change `sample_sizes`, `monte_carlo_count`, and `significance_levels` to fit the scope of your research. Just ensure the `sample_sizes` and `significance_levels` are consistent between the `critical_value` and `power` experiments.
+
+By following this structured, three-step approach, you can systematically evaluate and select the best statistical test for any given problem.
+
+---
+
+## How to Run This Example Workflow
+
+The `pysatl-experiment` command-line tool is designed to look for experiment configurations in a dedicated `.experiment` directory at the root of the project.
+
+Therefore, to run these examples, you must first copy the JSON template files into that location.
+
+**1. Copy the configuration files:**
+
+Firstly you need create files
+```bash
+poerty run experiment create <1_calculate_critical_values>
+```
+
+**2. Run the experiments in sequence:**
+
+Once the files are in the `.experiment` directory, you can execute the workflow steps in order. Make sure you are in the project's root directory.
+### First, run the calibration step
+```
+poetry run experiment build-and-run 1_calculate_critical_values
+```
+### Second, run the power evaluation
+```
+poetry run experiment build-and-run 2_evaluate_power
+```
+### Finally, run the performance analysis
+```
+poetry run experiment build-and-run 3_measure_time_complexity
+```

experiment_example/configs/1_calculate_critical_values.json

@@ -0,0 +1,36 @@
+{
+  "config": {
+    "name": "1_calculate_critical_values",
+    "experiment_type": "critical_value",
+    "hypothesis": "normal",
+    "criteria": [
+      {
+        "criterion_code": "KS",
+        "parameters": []
+      },
+      {
+        "criterion_code": "AD",
+        "parameters": []
+      }
+    ],
+    "executor_type": "standard",
+    "generator_type": "standard",
+    "report_builder_type": "standard",
+    "report_mode": "with-chart",
+    "run_mode": "reuse",
+    "monte_carlo_count": 1000,
+    "sample_sizes": [
+      200,
+      400,
+      600,
+      800,
+      1000
+    ],
+    "significance_levels": [
+      0.01,
+      0.05,
+      0.1
+    ],
+    "storage_connection": "/home/moveton/projects/prog/pysatl-experiment/.storage/result.sqlite"
+  }
+}

experiment_example/configs/2_evaluate_power.json

@@ -0,0 +1,45 @@
+{
+  "config": {
+    "name": "2_evaluate_power",
+    "experiment_type": "power",
+    "hypothesis": "normal",
+    "alternatives": [
+      {
+        "generator_name": "NORMALGENERATOR",
+        "parameters": [
+          0.5,
+          1
+        ]
+      }
+    ],
+    "criteria": [
+      {
+        "criterion_code": "KS",
+        "parameters": []
+      },
+      {
+        "criterion_code": "AD",
+        "parameters": []
+      }
+    ],
+    "executor_type": "standard",
+    "generator_type": "standard",
+    "report_builder_type": "standard",
+    "report_mode": "with-chart",
+    "run_mode": "reuse",
+    "monte_carlo_count": 1000,
+    "sample_sizes": [
+      200,
+      400,
+      600,
+      800,
+      1000
+    ],
+    "significance_levels": [
+      0.01,
+      0.05,
+      0.1
+    ],
+    "storage_connection": "/home/moveton/projects/prog/pysatl-experiment/.storage/result.sqlite"
+  }
+}

experiment_example/configs/3_measure_time_complexity.json

@@ -0,0 +1,36 @@
+{
+  "config": {
+    "name": "3_measure_time_complexity",
+    "experiment_type": "time_complexity",
+    "hypothesis": "normal",
+    "criteria": [
+      {
+        "criterion_code": "KS",
+        "parameters": []
+      },
+      {
+        "criterion_code": "AD",
+        "parameters": []
+      }
+    ],
+    "executor_type": "standard",
+    "generator_type": "standard",
+    "report_builder_type": "standard",
+    "report_mode": "with-chart",
+    "run_mode": "reuse",
+    "monte_carlo_count": 1000,
+    "sample_sizes": [
+      200,
+      400,
+      600,
+      800,
+      1000
+    ],
+    "significance_levels": [
+      0.01,
+      0.05,
+      0.1
+    ],
+    "storage_connection": "/home/moveton/projects/prog/pysatl-experiment/.storage/result.sqlite"
+  }
+}

experiment_example/reports/critical_values_report.pdf

@@ -25,6 +25,7 @@ rich = "==13.9.4"
 click = ">=8.2.1"
 dacite = "==1.9.2"
 line_profiler = "5.0.0"
+pydantic = "^2.11.9"
 pysatl-criterion = {path = "./pysatl_criterion"}
 
 [tool.poetry.group.dev.dependencies]

experiment_example/reports/power_report.pdf

@@ -1,7 +1,6 @@
 from click import BadParameter, argument, command
 
-from pysatl_experiment.cli.commands.common.common import get_experiment_config, read_experiment_data
-from pysatl_experiment.configuration.experiment_config.experiment_config import ExperimentConfig
+from pysatl_experiment.cli.commands.common.common import read_experiment_data
 from pysatl_experiment.configuration.experiment_data.experiment_data import ExperimentData
 from pysatl_experiment.experiment_new.experiment.experiment import Experiment
 from pysatl_experiment.experiment_new.experiment_steps.experiment_steps import ExperimentSteps
@@ -20,42 +19,34 @@ def build_and_run(name: str) -> None:
 
     :param name: name of the experiment.
     """
-
     experiment_exists = if_experiment_exists(name)
     if not experiment_exists:
         raise BadParameter(f"Experiment with name {name} does not exists.")
 
     experiment_data_dict = read_experiment_data(name)
-    experiment_config = get_experiment_config(experiment_data_dict)
-
     experiment_data = validate_build_and_run(experiment_data_dict)
-
-    experiment_type = experiment_config["experiment_type"]
-
-    experiment_steps = _build_experiment(experiment_data, experiment_type)
+    experiment_steps = _build_experiment(experiment_data)
 
     experiment = Experiment(experiment_steps)
-
     experiment.run_experiment()
 
 
-def _build_experiment(experiment_data: ExperimentData[ExperimentConfig], experiment_type: str) -> ExperimentSteps:
+def _build_experiment(experiment_data: ExperimentData) -> ExperimentSteps:
     """
-    Build experiment.
-
-    :param experiment_data: experiment data.
-    :param experiment_type: experiment type.
+    Build experiment from validated ExperimentData object.
 
+    :param experiment_data: validated experiment data.
     :return: experiment steps.
     """
-
     experiment_type_to_factory = {
         "power": PowerExperimentFactory,
         "critical_value": CriticalValueExperimentFactory,
         "time_complexity": TimeComplexityExperimentFactory,
     }
 
-    experiment_factory = experiment_type_to_factory[experiment_type]
+    experiment_type_str = experiment_data.config.experiment_type.value
+
+    experiment_factory = experiment_type_to_factory[experiment_type_str]
 
     experiment_steps = experiment_factory(experiment_data).create_experiment_steps()