Add api tests stateless (kubeflow#1242)

* Add api tests stateless

Signed-off-by: fege <[email protected]>

* Fix github action failures, lint and nox

Signed-off-by: fege <[email protected]>

* Add too_slow and remove code to force the model version id

Signed-off-by: fege <[email protected]>

* Add filter_too_much

Signed-off-by: fege <[email protected]>

* add hook to avoid Unsatisfiable

Signed-off-by: fege <[email protected]>

* remove variable

Signed-off-by: fege <[email protected]>

* add artifact_states

Signed-off-by: fege <[email protected]>

* Add example to schema

Signed-off-by: fege <[email protected]>

* Add example in src

Signed-off-by: fege <[email protected]>

* register strategy for string of int64

Signed-off-by: fege <[email protected]>

* sort imports

Signed-off-by: fege <[email protected]>

* modify case for problematic endpoints

Signed-off-by: fege <[email protected]>

* add more examples

Signed-off-by: fege <[email protected]>

* pin urllib

Signed-off-by: fege <[email protected]>

* exclude problematic endpoints and test with valid data

Signed-off-by: fege <[email protected]>

* Add gha to run the test and mark them with fuzz

Signed-off-by: fege <[email protected]>

* revert change in Makefile pushed by error

Signed-off-by: fege <[email protected]>

* skip test not marked with e2e or fuzz, trigger the fuzz on pr comment

Signed-off-by: fege <[email protected]>

* trigger with label

Signed-off-by: fege <[email protected]>

* correct the label name

Signed-off-by: fege <[email protected]>

* add types and semplify the if

Signed-off-by: fege <[email protected]>

* do not run e2e if test-fuzz label is added

Signed-off-by: fege <[email protected]>

* unpin urllib

Signed-off-by: fege <[email protected]>

* modify lock

Signed-off-by: fege <[email protected]>

* fix: remove proc on label

Signed-off-by: Alessio Pragliola <[email protected]>

---------

Signed-off-by: fege <[email protected]>
Signed-off-by: Alessio Pragliola <[email protected]>
Co-authored-by: Alessio Pragliola <[email protected]>

Author: fege

.github/workflows/python-tests.yml

@@ -13,7 +13,6 @@ on:
       - ".github/ISSUE_TEMPLATE/**"
       - ".github/dependabot.yml"
       - "docs/**"
-
 jobs:
   lint:
     name: ${{ matrix.session }}
@@ -221,6 +220,15 @@ jobs:
           kubectl port-forward service/distribution-registry-test-service 5001:5001 &
           sleep 2
           nox --python=${{ matrix.python }} --session=e2e -- --cov-report=xml
+      - name: Nox test fuzz (main only)
+        if: github.ref == 'refs/heads/main'
+        working-directory: clients/python
+        run: |
+          kubectl port-forward -n kubeflow service/model-registry-service 8080:8080 &
+          kubectl port-forward -n minio svc/minio 9000:9000 &
+          kubectl port-forward service/distribution-registry-test-service 5001:5001 &
+          sleep 2
+          nox --python=${{ matrix.python }} --session=fuzz
 
   docs-build:
     name: ${{ matrix.session }}

clients/python/.gitignore

@@ -7,3 +7,4 @@
 __pycache__/
 venv/
 .port-forwards.pid
+.hypothesis/

clients/python/Makefile

@@ -36,6 +36,16 @@ test-e2e: deploy-latest-mr deploy-local-registry deploy-test-minio
 	$(MAKE) test-e2e-cleanup
 	@exit $$STATUS
 
+.PHONY: test-fuzz
+test-fuzz: deploy-latest-mr deploy-local-registry deploy-test-minio
+	@echo "Starting test-fuzz"
+	poetry install --all-extras
+	@set -a; . ../../scripts/manifests/minio/.env; set +a; \
+	poetry run pytest --fuzz -v -s --hypothesis-show-statistics
+	@rm -f ../../scripts/manifests/minio/.env
+	$(MAKE) test-e2e-cleanup
+	@exit $$STATUS
+
 .PHONY: test-e2e-run
 test-e2e-run:
 	@echo "Ensuring all extras are installed..."

clients/python/README.md

@@ -345,6 +345,12 @@ Then you can run tests:
 make test test-e2e
 ```
 
+Then you can run fuzz tests:
+
+```bash
+make test-fuzz
+```
+
 ### Using Nox
 
 Common tasks, such as building documentation and running tests, can be executed using [`nox`](https://github.com/wntrblm/nox) sessions.

clients/python/noxfile.py

@@ -60,6 +60,7 @@ def tests(session: Session) -> None:
         "pytest-asyncio",
         "uvloop",
         "olot",
+        "schemathesis",
     )
     session.run(
         "pytest",
@@ -83,6 +84,7 @@ def e2e_tests(session: Session) -> None:
         "boto3",
         "olot",
         "uvloop",
+        "schemathesis",
     )
     try:
         session.run(
@@ -99,6 +101,22 @@ def e2e_tests(session: Session) -> None:
             session.notify("coverage", posargs=[])
 
 
+@session(name="fuzz", python=python_versions)
+def fuzz_tests(session: Session) -> None:
+    """Run the fuzzing tests."""
+    session.install(
+        ".",
+        "requests",
+        "pytest",
+        "uvloop",
+        "olot",
+        "schemathesis",
+    )
+    session.run(
+        "pytest",
+        "--fuzz",
+        "-rA",
+    )
 @session(python=python_versions[0])
 def coverage(session: Session) -> None:
     """Produce the coverage report."""