chore(benchmarks): add --artifacts and --configs options to perf-run-scenario (#14406)

The goal is to allow people to select which benchmark configs they want
to run, this allows people to skip unrelated/unnecessary configs if they
want to iterate quickly locally.

Since we added --configs, moved artifacts to --artifacts as well,
keeping the other positional arguments for now

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [ ] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Author: brettlangdon

benchmarks/base/run.py

@@ -64,5 +64,13 @@ def run(scenario_py, cname, cvars, output_dir):
     output_dir = sys.argv[1]
     print("Saving results to {}".format(output_dir))
     config = read_config("config.yaml")
+
+    # Filter configs if BENCHMARK_CONFIGS is set
+    benchmark_configs = os.environ.get("BENCHMARK_CONFIGS")
+    if benchmark_configs:
+        allowed_configs = set(c.strip() for c in benchmark_configs.split(","))
+        config = {k: v for k, v in config.items() if k in allowed_configs}
+        print("Filtering to configs: {}".format(", ".join(sorted(config.keys()))))
+
     for cname, cvars in config.items():
         run("scenario.py", cname, cvars, output_dir)

scripts/perf-run-scenario

@@ -5,7 +5,7 @@ SCRIPTNAME=$(basename $0)
 
 if [[ $# -lt 3 ]]; then
     cat << EOF
-Usage: ${SCRIPTNAME} <scenario> <version> <version> [artifacts]
+Usage: ${SCRIPTNAME} <scenario> <version> <version> [--configs config1,config2,...] [--artifacts path]
 
 Versions can be specified in the following formats:
     - "ddtrace==0.51.0" - to install a specific version from PyPI
@@ -16,7 +16,8 @@ Examples:
     ${SCRIPTNAME} span ddtrace==0.51.0 ddtrace==0.50.0
     ${SCRIPTNAME} span Datadog/[email protected] Datadog/[email protected]
     ${SCRIPTNAME} span ddtrace==2.8.4 .
-    ${SCRIPTNAME} span ddtrace==0.51.0 ddtrace==0.50.0 ./artifacts/
+    ${SCRIPTNAME} span ddtrace==0.51.0 ddtrace==0.50.0 --artifacts ./artifacts/
+    ${SCRIPTNAME} django_simple ddtrace==2.8.4 . --configs tracer,baseline
 
 EOF
     exit 1
@@ -45,10 +46,27 @@ DDTRACE_V2="$1"
 shift
 
 ARTIFACTS=""
-if [[ $# -gt 0 ]]; then
-    ARTIFACTS="$1"
-    shift
-fi
+CONFIGS=""
+
+# Parse remaining arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --configs)
+            CONFIGS="$2"
+            shift # past argument
+            shift # past value
+            ;;
+        --artifacts)
+            ARTIFACTS="$2"
+            shift # past argument
+            shift # past value
+            ;;
+        *)
+            echo "Unknown option: $1"
+            exit 1
+            ;;
+    esac
+done
 
 
 # Build scenario image
@@ -69,6 +87,7 @@ CMD+=" -v $(pwd):/src/ \
   -e CARGO_BUILD_JOBS=12 \
   -e DDTRACE_INSTALL_V1=$(expand_git_version $DDTRACE_V1) \
   -e DDTRACE_INSTALL_V2=$(expand_git_version $DDTRACE_V2) \
+  -e BENCHMARK_CONFIGS=${CONFIGS} \
   ${TAG}"
 
 $CMD