Switch context by test (#204)

# Summary

This PR helps in switching to a proper context/variant by specifying a
test mark or full task's evergreen url.

It solves a source of friction in our day-to-day workflows, which is
finding which context to switch to in order to run a particular test.
And by being able to switch context by specifying evergreen task's url
we're a step closer to having automation that reruns a failed script
locally.

## Current workflow
To find a context to run the test locally we often start from two
places:
1. Starting from Evergreen UI (easier): look at the evergreen variant in
the evergreen UI, copy it and run `make switch context=<copied variant>`
or use interactive fzf-powered `make switch` (without args), paste or
type the variant and press enter.
2. Starting from an e2e test file (harder): find a `@pytest.mark` in the
file, copy it, go to evergreen.yml, find a task group containing that
file, copy its name, then find a variant where that task group is used,
copy variant name, run `make switch context=<copied variant>`...

## Proposed workflow
`make switcht` is a new tool which is a supercharged `make switch`. It's
named similarly (typing `make switch` is kind of a muscle memory now)
and works in two modes:
1. With explicit test mark/url on `test=` argument: 
- `make switcht test=<test>`, where `<test>` is a pytest mark (e.g.
`e2e_replica_set`) - it will list all the variants that this test is
running in to be picked with `fzf`
- `make switcht test=<task url>`, where `<task url>` is a full url to
the evergreen test - it will switch directly to the variant from that
test
2. Without specifying test argument, which works in a similar
interactive way like `make switch`, but lists first all test files with
their marks to fuzzy-find the test, which then is used to list pick its
variants: `make switcht`

## Requirements 
It requires: 
- [fzf](https://github.com/junegunn/fzf) - for the interactive fuzzy
finder (we already use it for `make switch`)
- [ripgrep](https://github.com/BurntSushi/ripgrep) - to *quickly* list
all the pytest marks in all e2e test files
- Having evg credentials in env vars, e.g. in your
`scripts/dev/contexts/private-context`
```
EVERGREEN_USER=$(cat ~/.evergreen.yml | yq .user)
EVERGREEN_API_KEY=$(cat ~/.evergreen.yml | yq .api_key)
```

## Changes
Majority of the implementation is in two places:
- `scripts/dev/switch_context_by_test.sh` - it's mostly for `fzf`,
finding pytest marks using `rg`. It calls
scripts/python/find_test_variants.py
- scripts/python/find_test_variants.py - a new python script that lists
all the variants of a pytest's mark (`--task-name`) or a variant from a
full evg task url (`--task-url`)
- introduced scripts/python directory, to limit proliferation of python
scripts in random directories. Having them in one place makes it easier
to import/run them as modules
- refactored some evergreen API helpers into
scripts/python/evergreen_api.py
- some minor, random changes improving debuggability of bash scripts

## Proof of Work
Here are some examples for the most common use cases:

**With explicit test= parameter**
- Switch to one of test's context by pytest mark: `make switcht
test=e2e_replica_set`
![switcht with
mark](https://github.com/user-attachments/assets/f05b7cd3-d674-4cd4-9dc2-034eb9f37e23)
- Switch to context by test's url: `make switcht
test='https://spruce.mongodb.com/task/[...]/logs?execution=0'`
![switcht with
url](https://github.com/user-attachments/assets/2add88f1-9a45-4b47-bbda-3d75b86ca00e)

**Without any parameters (fully interactive)**
- Switch to context by fuzzy-searching a test first: `make switcht`
![switcht by fuzzy
search](https://github.com/user-attachments/assets/991cedcc-8663-4aa4-9f8b-9cc9f4af516a)
- Switch to context by pasting full file path into test picker: `make
switcht`
![switcht by pasting full
filepath](https://github.com/user-attachments/assets/860e344c-cbe5-4cf7-b48a-71907af49ab8)
- Switch to context by pasting full url into fzf picker: `make switcht`
![switcht by pasting
url](https://github.com/user-attachments/assets/b7c0f5c4-30d2-4e9f-abf1-8a7e2197f09a)

Afterwards, if the context needs to be regenerated again just run `make
switch` (without t) + enter - it will remember last switched context and
be selected first.

## Checklist
- [ ] Have you linked a jira ticket and/or is the ticket in the title?
- [ ] Have you checked whether your jira ticket required DOCSP changes?
- [ ] Have you checked for release_note changes?

## Reminder (Please remove this when merging)
- Please try to Approve or Reject Changes the PR, keep PRs in review as
short as possible
- Our Short Guide for PRs:
[Link](https://docs.google.com/document/d/1T93KUtdvONq43vfTfUt8l92uo4e4SEEvFbIEKOxGr44/edit?tab=t.0)
- Remember the following Communication Standards - use comment prefixes
for clarity:
  * **blocking**: Must be addressed before approval.
  * **follow-up**: Can be addressed in a later PR or ticket.
  * **q**: Clarifying question.
  * **nit**: Non-blocking suggestions.
  * **note**: Side-note, non-actionable. Example: Praise 
  * --> no prefix is considered a question

Author: lsierant

Makefile

@@ -62,6 +62,9 @@ precommit:
 switch:
 	@ scripts/dev/switch_context.sh $(context) $(additional_override)
 
+switcht:
+	@ scripts/dev/switch_context_by_test.sh $(test)
+
 # builds the Operator binary file and docker image and pushes it to the remote registry if using a remote registry. Deploys it to
 # k8s cluster
 operator: configure-operator build-and-push-operator-image

scripts/__init__.py

@@ -0,0 +1 @@
+# Makes 'scripts' a Python package.

scripts/dev/contexts/root-context

@@ -116,3 +116,8 @@ export MDB_SEARCH_COMMUNITY_VERSION
 
 export MDB_SEARCH_COMMUNITY_NAME="mongodb-search-community"
 export MDB_SEARCH_COMMUNITY_REPO_URL="quay.io/mongodb"
+
+
+if [[ ${MDB_BASH_DEBUG:-0} == 1 ]]; then
+  export PS4='+(${BASH_SOURCE}:${LINENO})[^$?]: ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
+fi

scripts/dev/launch_e2e.sh

@@ -2,6 +2,7 @@
 
 set -Eeou pipefail
 
+test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
 
 # The script launches e2e test. Note, that the Operator and necessary resources are deployed
 # inside the test
@@ -49,5 +50,3 @@ else
 fi
 
 title "E2e test ${test} is finished"
-
-

scripts/dev/set_env_context.sh

@@ -1,13 +1,14 @@
 #!/usr/bin/env bash
 
 set -Eeou pipefail
+test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
 
 # shellcheck disable=1091
 source scripts/funcs/errors
 
 script_name=$(readlink -f "${BASH_SOURCE[0]}")
 script_dir=$(dirname "${script_name}")
-context_file="${script_dir}/../../.generated/context.export.env"
+context_file="$(realpath "${script_dir}/../../.generated/context.export.env")"
 
 if [[ ! -f ${context_file} ]]; then
     fatal "File ${context_file} not found! Make sure to follow this guide to get started: https://wiki.corp.mongodb.com/display/MMS/Setting+up+local+development+and+E2E+testing#SettinguplocaldevelopmentandE2Etesting-GettingStartedGuide(VariantSwitching)"

scripts/dev/switch_context_by_test.sh

@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+
+set -Eeou pipefail
+test "${MDB_BASH_DEBUG:-0}" -eq 1 && set -x
+
+usage() {
+  echo "Switch context by passing the test (evergreen task name or full evergreen task URL)."
+  echo "If there is more than one variant running given test, then fzf picker is used."
+  echo "Usage: $0 <test>"
+  echo "  <test> is a task name from .evergreen.yml (e.g. 'e2e_search_community_basic') or a full Evergreen task URL."
+}
+
+source scripts/funcs/errors
+
+list_pytest_marks() {
+  rg -g '*.py' -o --no-line-number --no-heading --replace '$1' -m 1 \
+    '@(?:pytest\.)?mark\.(e2e_[a-zA-Z0-9_]+)' \
+    docker/mongodb-kubernetes-tests
+}
+
+pick_test_by_file_mark_or_task_url() {
+    if ! test_list="$(list_pytest_marks | sort -u)"; then
+      echo "Couldn't list pytest marks."
+      echo "${test_list}"
+      return 1
+    fi
+
+    test=$(fzf --print-query --header-first --with-nth '{2}: {1}' -d ':' --accept-nth 2 \
+      --header "Select file or task to find contexts where its used. You can paste full task's evergreen url here" <<< "${test_list}") || true
+    if [[ -z ${test} ]]; then
+      echo "Aborted selecting test"
+      return 1
+    fi
+
+    # test may contain one or two lines (file:mark or just mark/url)
+    number_of_selected_lines=$(wc -l <<< "${test}")
+    if [[ ${number_of_selected_lines} -eq 2 ]]; then
+      test="$(tail -n 1 <<< "${test}")"
+    elif [[ ${number_of_selected_lines} -gt 2 ]]; then
+      echo "Too many lines selected: ${test}"
+      return 1
+    fi
+
+    echo "${test}"
+}
+
+main() {
+  test="${1:-}"
+
+  if [[ -z ${test} ]]; then
+    test=$(pick_test_by_file_mark_or_task_url)
+    echo "Selected test: ${test}"
+  fi
+
+  if [[ "${test}" = *spruce.mongodb.com* ]]; then
+    find_variant_arg="--task-url"
+  else
+    find_variant_arg="--task-name"
+  fi
+
+  if ! contexts=$(scripts/evergreen/run_python.sh scripts/python/find_test_variants.py "${find_variant_arg}" "${test}"); then
+    echo "Couldn't find any test contexts running test: ${test}"
+    echo "${contexts}"
+    exit 1
+  fi
+
+  echo "Found contexts that are running test: ${test}"
+  echo "${contexts}"
+
+  selected_context="${contexts}"
+  if [[ $(wc -l <<< "${contexts}") -gt 1 ]]; then
+    if ! selected_context=$(fzf --header "${test} runs in multiple variants/contexts. Select one to switch context into." --header-first --layout=reverse <<< "${contexts}"); then
+      echo "Aborted selecting context"
+      exit 1
+    fi
+  fi
+
+  scripts/dev/switch_context.sh "${selected_context}"
+}
+
+main "$@"

scripts/evergreen/flakiness-report.py renamed to scripts/evergreen/flakiness_report.py

@@ -1,9 +1,9 @@
-import json
-import os
 import sys
 
 import requests
 
+from scripts.python.evergreen_api import get_evergreen_auth_headers
+
 EVERGREEN_API = "https://evergreen.mongodb.com/api"
 
 
@@ -18,16 +18,13 @@ def print_usage():
 
 
 def get_variants_with_retried_tasks() -> dict[str, list[dict]]:
-    evg_user = os.environ.get("EVERGREEN_USER", "")
-    api_key = os.environ.get("API_KEY", "")
-
-    if len(sys.argv) != 2 or evg_user == "" or api_key == "":
+    if len(sys.argv) != 2:
         print_usage()
-        exit(1)
+        raise RuntimeError("Exactly one argument (patch version number) must be provided")
 
     version = sys.argv[1]
+    headers = get_evergreen_auth_headers()
 
-    headers = {"Api-User": evg_user, "Api-Key": api_key}
     print("Fetching build variants...", file=sys.stderr)
     build_ids = requests.get(url=f"{EVERGREEN_API}/rest/v2/versions/{version}", headers=headers).json()
     build_statuses = [build_status for build_status in build_ids["build_variants_status"]]

scripts/funcs/checks

@@ -1,11 +1,5 @@
 #!/usr/bin/env bash
 
-pushd "${PWD}" > /dev/null || return
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-cd "${DIR}" || return
-source errors
-popd > /dev/null || return
-
 check_env_var() {
     local var_name="$1"
     local msg="$2"

scripts/funcs/kubernetes

@@ -2,16 +2,9 @@
 
 set -Eeou pipefail
 
-pushd "${PWD}" > /dev/null || return
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
-cd "${DIR}" || return
-# shellcheck source=scripts/funcs/checks
-source checks
-# shellcheck source=scripts/funcs/errors
-source errors
-# shellcheck source=scripts/funcs/printing
-source printing
-popd > /dev/null || return
+source scripts/funcs/checks
+source scripts/funcs/errors
+source scripts/funcs/printing
 
 ensure_namespace() {
     local namespace="${1}"

scripts/python/__init__.py

@@ -0,0 +1 @@
+# Makes 'scripts.python' a Python package.