Merge pull request #102 from Infisical/PLATFRM-128-more-tests-for-a-vertical-slice

improvement: add e2e test for relay and gateway

Author: fangpenlin

e2e/.env.sample

@@ -0,0 +1,47 @@
+# E2E Test Configuration
+# Copy this file to .env and update the values as needed.
+# The .env file is automatically loaded when running e2e tests.
+
+# =============================================================================
+# Required Configuration
+# =============================================================================
+
+# Path to the Infisical backend directory
+# This should point to the backend folder of the infisical repository
+# Example: /Users/your-username/workspace/infisical/backend
+INFISICAL_BACKEND_DIR=/path/to/infisical/backend
+
+# =============================================================================
+# Optional Configuration
+# =============================================================================
+
+# Dockerfile to use for building the backend service
+# Defaults to "Dockerfile.dev.fips" if not set
+# INFISICAL_BACKEND_DOCKERFILE=Dockerfile.dev.fips
+
+# Path to the CLI executable to use for testing
+# If not set, the test will look for ./infisical-merge in the e2e directory
+# This is only needed when using the "subprocess" run method
+# INFISICAL_CLI_EXECUTABLE=./infisical-merge
+
+# Default run method for CLI commands in tests
+# Valid values: "functionCall" (default) or "subprocess"
+# - functionCall: Calls CLI functions directly (better for IDE debugging, can collect stdout/stderr)
+#   Note: Since all commands run in the same process, make sure to properly reset any global state between test cases
+# - subprocess: Runs CLI as a separate process (simulates real user interaction, better isolation, can collect stdout/stderr)
+# CLI_E2E_DEFAULT_RUN_METHOD=subprocess
+
+# =============================================================================
+# Development & Debugging Options
+# =============================================================================
+
+# Disable the compose container cache
+# When enabled (default), tests will reuse existing containers to speed up development
+# Set to "1" to disable caching and always start fresh containers
+# CLI_E2E_DISABLE_COMPOSE_CACHE=1
+
+# Disable Ryuk container cleanup
+# When enabled, testcontainers will keep containers running after tests complete
+# This is useful for debugging backend issues
+# Set to "true" to disable automatic container cleanup
+# TESTCONTAINERS_RYUK_DISABLED=true

e2e/Makefile

@@ -1,27 +1,13 @@
-.PHONY: filter-api generate-client clean
-
-# Variables
-API_JSON := api.json
-ALLOWED_ENDPOINTS := allowed-endpoints.json
-FILTERED_API_JSON := api-filtered.json
-OPENAPI_CFG := openapi-cfg.yaml
-
-# Filter API JSON to only include allowed endpoints
-filter-api:
-	@echo "Filtering API JSON using allowed endpoints..."
-	@jq --slurpfile allowed $(ALLOWED_ENDPOINTS) \
-		'.paths |= with_entries(select(.key as $$k | $$allowed[0] | index($$k)))' \
-		$(API_JSON) > $(FILTERED_API_JSON)
-	@echo "Filtered API JSON written to $(FILTERED_API_JSON)"
+.PHONY: generate-code api.json
 
 # Generate OpenAPI client using oapi-codegen
-generate-client: filter-api
+generate-code:
 	@echo "Generating OpenAPI client..."
-	@go tool oapi-codegen -config $(OPENAPI_CFG) $(FILTERED_API_JSON)
+	@go generate
 	@echo "Client generated successfully"
 
-# Clean generated files
-clean:
-	@echo "Cleaning generated files..."
-	@rm -f $(FILTERED_API_JSON)
-	@echo "Clean complete"
+# Fetch API docs and format with jq
+api.json:
+	@echo "Fetching API docs..."
+	@set -o pipefail && curl -f http://localhost:4000/api/docs/json | jq . > api.json.tmp && mv api.json.tmp api.json
+	@echo "API docs saved to api.json"

e2e/README.md

@@ -11,25 +11,33 @@ Please feel free to reach out to @fangpenlin on the Slack channel if you encount
 
 The main subject of testing is a CLI executable program, i.e., the `infisical` command.
 Currently we support two approaches to test the CLI.
-The default run method is `subprocess`.
-You can set the `CLI_E2E_DEFAULT_RUN_METHOD` environment variable to change the default run method.
-For example:
+The default run method is `functionCall`.
+You can set the `CLI_E2E_DEFAULT_RUN_METHOD` environment variable to change the default run method, either in your `.env` file or as an environment variable:
 
 ```bash
-export CLI_E2E_DEFAULT_RUN_METHOD=functionCall
+# In .env file:
+CLI_E2E_DEFAULT_RUN_METHOD=subprocess
+
+# Or as environment variable:
+export CLI_E2E_DEFAULT_RUN_METHOD=subprocess
 ```
 
 ### The `subprocess` run method
 
 The most straightforward way to run a CLI command and test it is to start a new subprocess with the command executable, control the input by environment vars, stdin, args and see how it connects to the server and what it outputs (stdout and stderr).
 The benefit of this approach is that it simulates what a user really does and we can collect the stdout and stderr easily.
+It also provides better isolation between test cases with less side effects from previous test cases.
+For example, if a previous test case forgot to reset a global state, it won't affect subsequent test cases since each command runs in a fresh process.
 The drawback of this approach is that attaching a debugger to the CLI command requires some extra work (like finding the PID and attaching to it by using the PID from the debugger).
-This is the default run method if not specified.
 
 When using this run method, you need to make sure the executable is at the default location `./infisical-merge`.
-Otherwise, you can also specify the path to the executable by setting the `INFISICAL_CLI_EXECUTABLE` environment variable like this:
+Otherwise, you can also specify the path to the executable by setting the `INFISICAL_CLI_EXECUTABLE` environment variable, either in your `.env` file or as an environment variable:
 
 ```bash
+# In .env file:
+INFISICAL_CLI_EXECUTABLE=/path/to/infisical-merge
+
+# Or as environment variable:
 export INFISICAL_CLI_EXECUTABLE=/path/to/infisical-merge
 ```
 
@@ -42,11 +50,70 @@ go build .
 ### The `functionCall` run method
 
 The function call method calls the `main` command function directly from the e2e test case.
-The benefit is that you can attach a debugger directly to the CLI process without extra effort.
-The drawback is that currently we cannot collect stdout and stderr.
-Some extra efforts might be needed to update the CLI code to abstract the stdout and stderr output from logs to make it possible.
-In the meantime, since this is not available, we didn't set it as the default value.
+The benefit is that you can attach a debugger directly to the CLI process without extra effort, and we can collect stdout and stderr just like with the subprocess method.
 With this run method, since we are linking the e2e test build with the CLI as a library directly, there's no need to build the executable separately.
+This is the default run method if not specified.
+Note that since all commands run in the same process, if state wasn't cleared correctly from a previous command call, it may affect the next test case. Make sure to properly reset any global state between test cases when using this method.
+
+## Environment Variables Configuration
+
+E2E tests support loading environment variables from a `.env` file for easier configuration management. The `.env` file is automatically loaded when tests run using the [godotenv autoload feature](https://github.com/joho/godotenv#autoload) from the open source [godotenv library](https://github.com/joho/godotenv). The autoload happens when the test package is imported, so as long as you run tests from the `e2e` directory, the `.env` file in that directory will be automatically loaded.
+
+### Using a `.env` file
+
+1. Copy the sample file to create your `.env` file:
+
+   ```bash
+   cd e2e
+   cp .env.sample .env
+   ```
+
+2. Edit `.env` and set the required values (see `.env.sample` for detailed comments on each variable):
+
+   ```bash
+   # Required: Path to the Infisical backend directory
+   INFISICAL_BACKEND_DIR=/path/to/infisical/backend
+
+   # Optional: Other configuration variables
+   # CLI_E2E_DEFAULT_RUN_METHOD=functionCall
+   # INFISICAL_CLI_EXECUTABLE=./infisical-merge
+   ```
+
+3. Run your tests - the `.env` file will be automatically loaded:
+   ```bash
+   cd e2e
+   go test github.com/infisical/cli/e2e-tests/relay
+   ```
+
+**Note:** The `.env` file is git-ignored, so your local configuration won't be committed to the repository. See `.env.sample` for all available configuration options with detailed explanations.
+
+### Available Environment Variables
+
+The following environment variables can be set either in your `.env` file or as shell environment variables:
+
+#### Required Variables
+
+- **`INFISICAL_BACKEND_DIR`**: Path to the backend folder of the [infisical repository](https://github.com/infisical/infisical). This is required for tests to work.
+  - Example: `/Users/your-username/workspace/infisical/backend`
+
+#### Optional Variables
+
+- **`INFISICAL_BACKEND_DOCKERFILE`**: Dockerfile to use for building the backend service. Defaults to `"Dockerfile.dev.fips"` if not set.
+
+- **`INFISICAL_CLI_EXECUTABLE`**: Path to the CLI executable to use for testing. If not set, tests will look for `./infisical-merge` in the e2e directory. Only needed when using the "subprocess" run method.
+
+- **`CLI_E2E_DEFAULT_RUN_METHOD`**: Default run method for CLI commands in tests. Valid values:
+
+  - `"functionCall"` (default): Calls CLI functions directly - better for IDE debugging, can collect stdout/stderr
+  - `"subprocess"`: Runs CLI as a separate process - simulates real user interaction, can collect stdout/stderr
+
+- **`CLI_E2E_DISABLE_COMPOSE_CACHE`**: Set to `"1"` to disable the compose container cache. When enabled (default), tests reuse existing containers to speed up development.
+
+- **`CLI_E2E_REMOVE_COMPOSE`**: Set to `"1"` to enable cleanup of compose containers after tests complete. When not set (default), containers will remain running after tests finish, which is useful for debugging. Our cache system will also reuse the existing container if matches, to speed up the development cycle. This controls whether the `t.Cleanup()` function removes the compose stack.
+
+- **`TESTCONTAINERS_RYUK_DISABLED`**: Set to `"true"` to disable Ryuk container cleanup. Useful for debugging backend issues as containers will remain running after tests complete.
+
+For detailed descriptions and examples, see the `.env.sample` file.
 
 ## Setting the `INFISICAL_BACKEND_DIR` value
 
@@ -62,8 +129,7 @@ infisical := NewInfisicalService().
 ```
 
 Because it runs the actual Infisical API server stack, you need to specify the `INFISICAL_BACKEND_DIR` value and point it to the `backend` folder of the [infisical repository](https://github.com/infisical/infisical) to make it work.
-For example, if you have the repo checked out at `/Users/fangpen/workspace/infisical`.
-Then you can set the environment variable like this:
+The easiest way is to set it in your `.env` file (see above), or you can export it as an environment variable:
 
 ```bash
 export INFISICAL_BACKEND_DIR=/Users/fangpen/workspace/infisical/backend
@@ -85,7 +151,14 @@ cd e2e
 go test github.com/infisical/cli/e2e-tests/relay
 ```
 
-Combining the exported environment variables, you might end up with running commands like this:
+If you're using a `.env` file (recommended), just make sure it's configured and run the tests:
+
+```bash
+cd e2e
+go test github.com/infisical/cli/e2e-tests/relay
+```
+
+Alternatively, you can export environment variables manually:
 
 ```bash
 export INFISICAL_CLI_EXECUTABLE=/path/to/infisical-merge
@@ -94,7 +167,7 @@ cd e2e
 go test github.com/infisical/cli/e2e-tests/relay
 ```
 
-It's a bit verbose right now, but we will improve the quality of life over time by adding things such as a Makefile to make it much easier.
+**Tip:** Using a `.env` file is much more convenient than exporting variables manually. See the [Environment Variables Configuration](#environment-variables-configuration) section above for details.
 
 ## Troubleshooting the failing tests due to CLI error
 
@@ -116,20 +189,26 @@ less /var/folders/wc/g97rf4092_z9wqbp93djvnt00000gn/T/TestRelay_RegistersARelay2
 
 Then you should be able to find out why it fails.
 You can also switch the call method to `functionCall` and set up a debugger to trace into the CLI program to find out why it fails.
-If you run the command test with `functionCall`, it will not write the stdout / stderr to a file, but instead, it should print it to the console where you run the tests.
+With the `functionCall` method, the stdout and stderr logs are also written to temp files, just like with the `subprocess` method, so you can inspect them in the same way.
 
 ## Troubleshooting the failing tests due to Infisical backend API errors
 
 If the errors happen in the backend, to find out what's going on, you can open the Docker Desktop app or use `docker ps` and then `docker logs` to find out what the error message is in the Infisical backend API server.
 Please note that by default, the [testcontainers library](https://github.com/testcontainers/testcontainers-go) (the library we use to run the docker-compose for the Infisical stack) will start a container called Ryuk for deleting the containers after the test is finished.
 Because of that, if you run into an error in the backend reproduced by running the test, the container might already be gone after the test finishes.
 Then you won't be able to look inside the container and find out what's going on.
-To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment variable to `true` like this to disable the container deleting behavior:
+To solve the problem, you can set the `TESTCONTAINERS_RYUK_DISABLED` environment variable to `true` to disable the container deleting behavior. You can do this either in your `.env` file:
 
 ```bash
 TESTCONTAINERS_RYUK_DISABLED=true
 ```
 
+Or as an environment variable:
+
+```bash
+export TESTCONTAINERS_RYUK_DISABLED=true
+```
+
 To learn more about the behavior of Ryuk from testcontainers, please read [their document here](https://golang.testcontainers.org/features/garbage_collector/#ryuk).
 
 ## Use compose containers cache to speed up the development cycle
@@ -151,8 +230,57 @@ If you have `TESTCONTAINERS_RYUK_DISABLED` set to `true`, each time the `Up` met
 If there's such a container already running, we will reuse it by resetting its database instead of starting a new one.
 That way, it's much faster than booting up a new compose stack and waiting for it to get online.
 
-If for any reason the cache system is not working as desired, you can disable it by setting the `CLI_E2E_DISABLE_COMPOSE_CACHE` value to `1` like this:
+If for any reason the cache system is not working as desired, you can disable it by setting the `CLI_E2E_DISABLE_COMPOSE_CACHE` value to `1`. You can do this either in your `.env` file:
+
+```bash
+CLI_E2E_DISABLE_COMPOSE_CACHE=1
+```
+
+Or as an environment variable:
 
 ```bash
 export CLI_E2E_DISABLE_COMPOSE_CACHE=1
 ```
+
+## Generate Infisical client code
+
+The e2e tests use generated client code to interact with the Infisical backend API.
+The client code is generated from the OpenAPI specification using `go:generate` and the `oapi-codegen` tool.
+
+### How it works
+
+1. **Download the OpenAPI specification**: While the Infisical backend server is running (typically on `http://localhost:4000`), download the OpenAPI JSON specification:
+
+   You can use the Makefile target to fetch and format the API docs:
+
+   ```bash
+   cd e2e
+   make api.json
+   ```
+
+   This will fetch the OpenAPI spec from `http://localhost:4000/api/docs/json`, format it with `jq`, and save it to `api.json`. If the request fails, the existing `api.json` file will not be overwritten and error messages will be displayed.
+
+   Alternatively, you can use curl directly:
+
+   ```bash
+   curl http://localhost:4000/api/docs/json -o api.json
+   ```
+
+2. **Important note**: Before downloading the OpenAPI spec, you may need to manually set `hidden: false` for some internal endpoints in the backend code. This step is currently done manually, but a CI job should automate this in the future.
+
+3. **Generate the client code**: Once you have the `api.json` file in the `e2e` folder, you can generate the client code using either:
+
+   - `go generate` command:
+     ```bash
+     cd e2e
+     go generate
+     ```
+   - Or the Makefile target:
+     ```bash
+     cd e2e
+     make generate-code
+     ```
+
+   This will generate `packages/client/client.gen.go` from the OpenAPI specification using the configuration in `openapi-cfg.yaml`.
+
+The `go:generate` directive is defined in `e2e/main.go` and uses the `oapi-codegen` tool to generate type-safe Go client code from the OpenAPI specification.