V2 Framework draft (#1236)

* V2 Framework PoC

Author: skudasov

framework/.gitignore

@@ -0,0 +1,34 @@
+### Go template
+# If you prefer the allow list template instead of the deny list, see community template:
+# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
+#
+# Binaries for programs and plugins
+*.exe
+*.exe~
+*.dll
+*.so
+*.dylib
+
+# Test binary, built with `go test -c`
+*.test
+
+# Output of the go coverage tool, specifically when used with LiteIDE
+*.out
+
+# Dependency directories (remove the comment below to include it)
+# vendor/
+
+# Go workspace file
+go.work
+
+.envrc
+promtail-config.yml
+
+# CL node
+components/clnode/config.toml
+components/clnode/secrets.toml
+components/clnode/overrides.toml
+components/clnode/secrets-overrides.toml
+components/clnode/user-overrides.toml
+components/clnode/user-secrets-overrides.toml
+

framework/CACHING.md

@@ -0,0 +1,28 @@
+## Component caching
+
+We use component caching to accelerate test development and enforce idempotent test actions.
+
+Each component is designed to be pure with defined inputs/outputs.
+
+You can use an environment variable to skip deployment steps and use cached outputs if your infrastructure is already running (locally or remotely):
+
+```
+export CTF_CONFIGS=smoke-cache.toml
+export CTF_USE_CACHED_OUTPUTS=true
+```
+
+### Using remote components
+
+Because components are decoupled through outputs, you can use a cached config and switch outputs to any deployed infrastructure, such as staging. This allows you to reuse the same testing logic for behavior validation.
+
+Example:
+```
+[blockchain_a.out]
+chain_id = '31337'
+
+[[blockchain_a.out.nodes]]
+ws_url = 'ws://127.0.0.1:33447'
+http_url = 'http://127.0.0.1:33447'
+docker_internal_ws_url = 'ws://anvil-3716a:8900'
+docker_internal_http_url = 'http://anvil-3716a:8900'
+```

framework/COMPONENTS.md

@@ -0,0 +1,74 @@
+## Developing Components
+
+To build a scalable framework that enables the reuse of our product deployments (contracts or services in Docker), we need to establish a clear component structure.
+```
+package mycomponent
+
+import (
+	"fmt"
+	"github.com/smartcontractkit/chainlink-testing-framework/framework"
+)
+
+type Input struct {
+    // inputs fields that component exposes for configuration
+    ...
+    // outputs are embedded into inputs so framework can automatically save them
+	Out                      *Output  `toml:"out"`
+}
+
+type Output struct {
+    // outputs that will be dumped to config and cached
+}
+
+
+func NewComponent(input *Input) (*Output, error) {
+	if input.Out != nil && framework.UseCache() {
+		return input.Out, nil
+	}
+	
+	// component logic here
+	// deploy a docker container(s)
+	// or deploy a set of smart contracts
+	
+	input.Out = &Output{...}
+	return out, nil
+}
+```
+Each component can define inputs and outputs, following these rules:
+
+- Outputs should be included within inputs.
+- If your component is used for side effects output can be omitted.
+- `if input.Out != nil && framework.UseCache()` should be added if you'd like to use caching
+
+### Docker components good practices for [testcontainers-go](https://golang.testcontainers.org/):
+- `ContainerRequest` must contain labels, network and alias required for local observability stack and deployment isolation
+```
+		Labels:   framework.DefaultTCLabels(),
+		Networks: []string{framework.DefaultNetworkName},
+		NetworkAliases: map[string][]string{
+			framework.DefaultNetworkName: {containerName},
+		},
+```
+- In order to copy files into container use `framework.WriteTmpFile(data string, fileName string)`
+```
+	userSecretsOverridesFile, err := WriteTmpFile(in.Node.UserSecretsOverrides, "user-secrets-overrides.toml")
+	if err != nil {
+		return nil, err
+	}
+```
+- Output of docker component must contain all the URLs component exposes for access, both for internal docker usage and external test (host) usage
+```
+	host, err := framework.GetHost(c)
+	if err != nil {
+		return nil, err
+	}
+	mp, err := c.MappedPort(ctx, nat.Port(bindPort))
+	if err != nil {
+		return nil, err
+	}
+
+	return &NodeOut{
+		DockerURL: fmt.Sprintf("http://%s:%s", containerName, in.Node.Port),
+		HostURL:   fmt.Sprintf("http://%s:%s", host, mp.Port()),
+	}, nil
+```

framework/CONFIGURATION.md

@@ -0,0 +1,55 @@
+## Test Configuration
+
+In our framework, components control the configuration. Each component defines an Input that you embed into your test configuration. We automatically ensure consistency between TOML and struct definitions during validation.
+
+An example of your component configuration:
+```
+// Input is a blockchain network configuration params declared by the component
+type Input struct {
+	Type                     string   `toml:"type" validate:"required,oneof=anvil geth" envconfig:"net_type"`
+	Image                    string   `toml:"image" validate:"required"`
+	Tag                      string   `toml:"tag" validate:"required"`
+	Port                     string   `toml:"port" validate:"required"`
+	ChainID                  string   `toml:"chain_id" validate:"required"`
+	DockerCmdParamsOverrides []string `toml:"docker_cmd_params"`
+	Out                      *Output  `toml:"out"`
+}
+```
+
+How you use it in tests:
+```
+type Config struct {
+	BlockchainA        *blockchain.Input `toml:"blockchain_a" validate:"required"`
+}
+
+func TestDON(t *testing.T) {
+	in, err := framework.Load[Config](t)
+	require.NoError(t, err)
+
+	// deploy component
+	bc, err := blockchain.NewBlockchainNetwork(in.BlockchainA)
+	require.NoError(t, err)
+```
+
+In `TOML`:
+```
+[blockchain_a]
+chain_id = "31337"
+docker_cmd_params = ["--block-time=1"]
+image = "f4hrenh9it/foundry"
+port = "8500"
+tag = "latest"
+type = "anvil"
+```
+
+### Best practices for configuration and validation
+- Avoid stateful types (e.g., loggers, clients) in your config.
+- All `input` fields should include validate: "required", ensuring consistency between TOML and struct definitions.
+- Add extra validation rules for URLs or "one-of" variants. Learn more here: go-playground/validator.
+
+### Overriding configuration
+To override any configuration, we merge multiple files into a single struct.
+
+You can specify multiple file paths using `CTF_CONFIGS=path1,path2,path3`.
+
+The framework will apply these configurations from right to left.

framework/README.md

@@ -0,0 +1,32 @@
+## Chainlink Testing Framework Harness
+
+<!-- TOC -->
+* [CLI](./cmd/README.md)
+* [Components](./COMPONENTS.md)
+* [Configuration](./CONFIGURATION.md)
+* [Caching](./CACHING.md)
+* [Local Observability Stack](./cmd/observability/README.md)
+* [Examples](https://github.com/smartcontractkit/chainlink/tree/8e8597aa14c39c48ed4b3261f6080fa43b5d7cd0/e2e/capabilities)
+<!-- TOC -->
+
+This module includes the CTFv2 harness, a lightweight, modular, and data-driven framework designed for combining off-chain and on-chain components while implementing best practices for end-to-end system-level testing:
+
+- **Non-nil configuration**: All test variables must have defaults, automatic validation.
+- **Component isolation**: Components are decoupled via input/output structs, without exposing internal details.
+- **Modular configuration**: No arcane knowledge of framework settings is required; the config is simply a reflection of the components being used in the test. Components declare their own configuration—'what you see is what you get.'
+- **Replaceability and extensibility**: Since components are decoupled via outputs, any deployment component can be swapped with a real service without altering the test code.
+- **Integrated observability stack**: use `ctf obs up` to spin up a local observability stack.
+- **Connectivity**: seamless connection of production-ready components and local components using [testcontainers-go networking](https://golang.testcontainers.org/features/networking/#exposing-host-ports-to-the-container)."
+
+
+### Environment variables (Tests, when using in Go code)
+|             Name             |                                                                      Description                                                                       |          Possible values | Default |        Required?         |
+|:----------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------:|-------------------------:|:-------:|:------------------------:|
+|         CTF_CONFIGS          | Path(s) to test config files. <br/>Can be more than one, ex.: smoke.toml,smoke_1.toml,smoke_2.toml.<br/>First filepath will hold all the merged values | Any valid TOML file path |         |            ✅             |
+|        CTF_LOG_LEVEL         |                                                                   Harness log level                                                                    | `info`, `debug`, `trace` | `info`  |            🚫            |
+|       CTF_LOKI_STREAM        |                                                Streams all components logs to `Loki`, see params below                                                 |          `true`, `false` | `false` |            🚫            |
+|           LOKI_URL           |                                            URL to `Loki` push api, should be like`${host}/loki/api/v1/push`                                            |                      URL |    -    | If you use `Loki` then ✅ |
+|        LOKI_TENANT_ID        |                                                Streams all components logs to `Loki`, see params below                                                 |          `true`, `false` |    -    | If you use `Loki` then ✅ |
+| TESTCONTAINERS_RYUK_DISABLED |                                   Testcontainers-Go reaper container, removes all the containers after the test exit                                   |          `true`, `false` | `false` |            🚫            |
+|    CTF_USE_CACHED_OUTPUTS    |                                                  Use cached `.out` fields from `*-cache.toml` config                                                   |          `true`, `false` | `false` |            🚫            |
+|         RESTY_DEBUG          |                                                            Log all Resty client HTTP calls                                                             |          `true`, `false` | `false` |            🚫            |