Extended compare and run-remote tool (#452)

* compare improvements

* added grafana uid

* feat: enhance run-remote robustness and reliability

• Fixed massive variance blocking regression detection - Added proper variance-based validation that prevents false positives when latency data is too unstable for reliable analysis

• Fixed missing test links in performance summaries - Corrected regression counting logic so clickable Grafana links appear in bullet lists for confirmed regressions

• Added conditional AWS credentials checking - Credentials are only required when actually needed (for infrastructure deployment or S3 upload, not when using existing infrastructure with --inventory)

• Added automatic tool installation - Missing dependencies like redis-server, zip, and memtier_benchmark are automatically installed on remote nodes using official Redis APT repository

• Enhanced S3 upload error handling - Upload failures don't crash the process but continue with warnings, ensuring benchmark results are preserved locally even when S3 is unavailable

These changes make run-remote much more robust, user-friendly, and reliable by eliminating false regression alerts, providing better diagnostic information, reducing setup requirements, and gracefully handling common failure scenarios.

* Fixed black/flake8 issues

* Added test for specific oss cluster setup name

* fixes per PR failures

* Fixed mock server key error

* Extended run-local tests with dry run validation. Ensured setup names are store in timeseries. fixed tox-docker deps

* Fixed rts usage on tests

* Fixed common tests

* fixed import os error

* fixed run_remote

---------

Co-authored-by: fcostaoliveira <[email protected]>

Author: filipecosta90

Makefile

@@ -0,0 +1,45 @@
+# Makefile for redisbench-admin
+
+.PHONY: compliance compliance-fix test integration-tests help
+
+# Code quality and compliance checks
+compliance:
+	@echo "🔍 Running compliance checks..."
+	tox -e compliance
+
+# Fix code formatting issues
+format:
+	@echo "🔧 Fixing code formatting..."
+	tox -e format
+	@echo "✅ Code formatting fixed!"
+
+# Alias for format
+compliance-fix: format
+
+# Run tests with coverage
+test:
+	@echo "🧪 Running tests..."
+	poetry run coverage erase
+	poetry run pytest --cov=redisbench_admin --cov-report=term-missing -ra
+	poetry run coverage xml
+	@echo "✅ Tests completed!"
+
+# Run integration tests (alias for test)
+integration-tests: test
+
+# Run both compliance and tests
+all: compliance test
+
+# Show help
+help:
+	@echo "Available targets:"
+	@echo "  compliance       - Run code quality checks (black, flake8)"
+	@echo "  format          - Fix code formatting with black"
+	@echo "  compliance-fix   - Alias for format"
+	@echo "  test            - Run tests with coverage"
+	@echo "  integration-tests - Alias for test"
+	@echo "  all             - Run compliance checks and tests"
+	@echo "  help            - Show this help message"
+
+# Default target
+.DEFAULT_GOAL := help

README.md

@@ -110,7 +110,13 @@ $ tox
 
 To run a specific test:
 ```sh
-$ tox -- tests/test_redistimeseries.py
+$ tox -- tests/test_defaults_purpose_built_env.py
+```
+
+To run a specific test and persist the docker container used for timeseries:
+
+```
+tox  --docker-dont-stop=rts_datasink  -- -vv --log-cli-level=INFO  tests/test_defaults_purpose_built_env.py 
 ```
 
 To run a specific test with verbose logging:

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "redisbench-admin"
-version = "0.11.37"
+version = "0.11.40"
 description = "Redis benchmark run helper. A wrapper around Redis and Redis Modules benchmark tools ( ftsb_redisearch, memtier_benchmark, redis-benchmark, aibench, etc... )."
 authors = ["filipecosta90 <[email protected]>","Redis Performance Group <[email protected]>"]
 readme = "README.md"
@@ -44,6 +44,9 @@ certifi = ">=2021.10.8,<2025.0.0"
 pygithub = "^1.57"
 
 [tool.poetry.dev-dependencies]
+tox = ">=4.16.0"
+tox-docker = ">=5.0.0"
+docker = ">=7.1.0"
 pytest = "^4.6"
 pytest-cov = "^2.9.0"
 codecov = "2.1.13"

redisbench_admin/compare/args.py

@@ -135,7 +135,7 @@ def create_compare_arguments(parser):
     parser.add_argument(
         "--regressions-percent-lower-limit",
         type=float,
-        default=5.0,
+        default=8.0,
         help="Only consider regressions with a percentage over the defined limit. (0-100)",
     )
     parser.add_argument(
@@ -162,6 +162,10 @@ def create_compare_arguments(parser):
         type=str,
         default="https://benchmarksrediscom.grafana.net/d/",
     )
+    parser.add_argument(
+        "--grafana_uid",
+        default=None,
+    )
     parser.add_argument(
         "--auto-approve",
         required=False,