Skip to content

Commit a7eb48b

Browse files
Separate configuration, data, state and runtime directories (#484)
1 parent cab252c commit a7eb48b

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+1620
-525
lines changed

.github/workflows/commit.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,8 @@ jobs:
6060

6161
- name: "Cache Envoy binaries"
6262
uses: actions/cache@v4
63-
with: # ~/.func-e/versions is cached so that we only re-download once: for TestFuncEInstall
64-
path: ~/.func-e/versions
63+
with: # ~/.local/share/func-e/envoy-versions is cached so that we only re-download once: for TestFuncEInstall
64+
path: ~/.local/share/func-e/envoy-versions
6565
key: test-${{ runner.os }}-envoy-${{ hashFiles('internal/version/last_known_envoy.txt') }}
6666
restore-keys: test-${{ runner.os }}-envoy-
6767

CONFIG.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# func-e configuration
2+
3+
The [XDG Base Directory Specification][xdg] defines standard locations for
4+
user-specific files:
5+
6+
- **Data files**: Persist across sessions, e.g. downloaded binaries
7+
- **State files**: Persist between restarts but non-portable, e.g. stdout.log
8+
- **Runtime files**: Ephemeral files, e.g. admin-address.txt
9+
10+
func-e adopts these conventions to separate downloaded Envoy binaries, logs,
11+
and ephemeral admin addresses. Doing so allows library consumers like Envoy AI
12+
Gateway to define their own home directories under an XDG base convention.
13+
14+
## Configuration mappings
15+
16+
| Environment Variable | Default Path | API Option |
17+
|-----------------------|-------------------------|---------------------|
18+
| `FUNC_E_CONFIG_HOME` | `~/.config/func-e` | `api.ConfigHome()` |
19+
| `FUNC_E_DATA_HOME` | `~/.local/share/func-e` | `api.DataHome()` |
20+
| `FUNC_E_STATE_HOME` | `~/.local/state/func-e` | `api.StateHome()` |
21+
| `FUNC_E_RUNTIME_DIR` | `/tmp/func-e-${UID}` | `api.RuntimeDir()` |
22+
| `FUNC_E_RUN_ID` | auto-generated | `api.RunID()` |
23+
24+
| File Type | Purpose | Default Path |
25+
|------------------------|----------------------------------------------|----------------------------------------------------------------------------------|
26+
| Selected Envoy Version | Version preference (persistent, shared) | `${FUNC_E_CONFIG_HOME}/envoy-version` |
27+
| Envoy Binaries | Downloaded executables (persistent, shared) | `${FUNC_E_DATA_HOME}/envoy-versions/{version}/bin/envoy` |
28+
| Envoy Run State | Per-run logs & config (persistent debugging) | `${FUNC_E_STATE_HOME}/envoy-runs/{runID}/stdout.log,stderr.log,config_dump.json` |
29+
| Admin Address Default | Generated endpoint (ephemeral, per-run) | `${FUNC_E_RUNTIME_DIR}/{runID}/admin-address.txt` |
30+
31+
- **Correlation ID (`runID`)**: `YYYYMMDD_HHMMSS_UUU`
32+
- (epoch date, time, last 3 digits of micros to allow concurrent runs)
33+
- Can be customized via `FUNC_E_RUN_ID` environment variable or `--run-id` flag
34+
- Custom runID cannot contain path separators (/ or \)
35+
- Example: `--run-id 0` for predictable Docker/Kubernetes deployments
36+
- **Directory per run** isolates concurrent runs and ensures correlation
37+
38+
## Legacy Mapping
39+
40+
**Deprecation Warning**:
41+
```
42+
WARNING: FUNC_E_HOME is deprecated and will be removed in a future version.
43+
Please migrate to FUNC_E_CONFIG_HOME, FUNC_E_DATA_HOME, FUNC_E_STATE_HOME or FUNC_E_RUNTIME_DIR.
44+
```
45+
46+
| File Type | Legacy Path Pattern | XDG Path Pattern |
47+
|------------------------|-----------------------------------------------|-------------------------------------------------------------|
48+
| Selected Envoy Version | `$FUNC_E_HOME/version` | `$FUNC_E_CONFIG_HOME/envoy-version` |
49+
| Envoy Binaries | `$FUNC_E_HOME/versions/{version}/bin/envoy` | `$FUNC_E_DATA_HOME/envoy-versions/{version}/bin/envoy` |
50+
| Run Logs | `$FUNC_E_HOME/runs/{epoch}/stdout.log` | `$FUNC_E_STATE_HOME/envoy-runs/{runID}/stdout.log` |
51+
| Admin Address | `$FUNC_E_HOME/runs/{epoch}/admin-address.txt` | `$FUNC_E_RUNTIME_DIR/{runID}/admin-address.txt` |
52+
53+
These legacy patterns will be supported only when `FUNC_E_HOME` is set and will
54+
be removed in a future version. A file envoy.pid will not be written as it
55+
isn't necessary.
56+
57+
---
58+
[xdg]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

RATIONALE.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,3 +23,31 @@ much an emergency as if we put emulation in the critical path (ex in PR tests.)
2323

2424
At the point when GitHub Actions supports free arm64 runners, we can simplify by removing Travis.
2525
Azure DevOps Pipelines already supports arm64, so it is possible GitHub Actions will in the future.
26+
27+
## Why do we have so many environment variables for file locations?
28+
29+
Recently, func-e is used as a library, which means its default locations need
30+
to support layouts that are non-default. We also want to be able to predict the
31+
log directory when running in Docker instead of using a timestamp in the path.
32+
33+
This results in several config locations like this:
34+
35+
| Environment Variable | Default Path | API Option |
36+
|-----------------------|-------------------------|---------------------|
37+
| `FUNC_E_CONFIG_HOME` | `~/.config/func-e` | `api.ConfigHome()` |
38+
| `FUNC_E_DATA_HOME` | `~/.local/share/func-e` | `api.DataHome()` |
39+
| `FUNC_E_STATE_HOME` | `~/.local/state/func-e` | `api.StateHome()` |
40+
| `FUNC_E_RUNTIME_DIR` | `/tmp/func-e-${UID}` | `api.RuntimeDir()` |
41+
| `FUNC_E_RUN_ID` | auto-generated | `api.RunID()` |
42+
43+
These are conventional to [XDG][xdg], which makes it easier to explain to
44+
people. Also, XDS conventions are used by Prometheus and block/goose, so will
45+
be familiar to some.
46+
47+
In summary, XDS conventions allow dependents like Envoy AI Gateway to brand
48+
their own directories and co-mingle its configuration and logs with those
49+
of func-e when it runs Envoy (the gateway process). It also allows Docker to
50+
export `FUNC_E_RUN_ID=0` to aid in location of key files.
51+
52+
---
53+
[xdg]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

USAGE.md

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,18 +4,28 @@ downloads and installs the latest version of Envoy for you.
44

55
To list versions of Envoy you can use, execute `func-e versions -a`. To
66
choose one, invoke `func-e use 1.35.3`. This installs into
7-
`$FUNC_E_HOME/versions/1.35.3`, if not already present. You may also use
8-
minor version, such as `func-e use 1.35`.
7+
`$FUNC_E_DATA_HOME/envoy-versions/1.35.3`, if not already present. You may
8+
also use minor version, such as `func-e use 1.35`.
99

1010
You may want to override `$ENVOY_VERSIONS_URL` to supply custom builds or
1111
otherwise control the source of Envoy binaries. When overriding, validate
1212
your JSON first: https://archive.tetratelabs.io/release-versions-schema.json
1313

14+
Directory structure:
15+
`$FUNC_E_CONFIG_HOME` stores configuration files
16+
(default: ${HOME}/.config/func-e)
17+
`$FUNC_E_DATA_HOME` stores Envoy binaries
18+
(default: ${HOME}/.local/share/func-e)
19+
`$FUNC_E_STATE_HOME` stores logs
20+
(default: ${HOME}/.local/state/func-e)
21+
`$FUNC_E_RUNTIME_DIR` stores temporary files
22+
(default: /tmp/func-e-${UID})
23+
1424
Advanced:
1525
`FUNC_E_PLATFORM` overrides the host OS and architecture of Envoy binaries.
1626
This is used when emulating another platform, e.g. x86 on Apple Silicon M1.
1727
Note: Changing the OS value can cause problems as Envoy has dependencies,
18-
such as glibc. This value must be constant within a `$FUNC_E_HOME`.
28+
such as glibc. This value must be constant within a `$FUNC_E_DATA_HOME`.
1929

2030
# Commands
2131

@@ -32,6 +42,11 @@ such as glibc. This value must be constant within a `$FUNC_E_HOME`.
3242

3343
| Name | Usage | Default |
3444
| ---- | ----- | ------- |
35-
| FUNC_E_HOME | func-e home directory (location of installed versions and run archives) | ${HOME}/.func-e |
45+
| FUNC_E_HOME | (deprecated) func-e home directory - use --config-home, --data-home, --state-home or --runtime-dir instead | |
46+
| FUNC_E_CONFIG_HOME | directory for configuration files | ${HOME}/.config/func-e |
47+
| FUNC_E_DATA_HOME | directory for Envoy binaries | ${HOME}/.local/share/func-e |
48+
| FUNC_E_STATE_HOME | directory for logs (used by run command) | ${HOME}/.local/state/func-e |
49+
| FUNC_E_RUNTIME_DIR | directory for temporary files (used by run command) | /tmp/func-e-${UID} |
50+
| FUNC_E_RUN_ID | custom run identifier for logs/runtime directories (used by run command) | auto-generated timestamp |
3651
| ENVOY_VERSIONS_URL | URL of Envoy versions JSON | https://archive.tetratelabs.io/envoy/envoy-versions.json |
3752
| FUNC_E_PLATFORM | the host OS and architecture of Envoy binaries. Ex. darwin/arm64 | $GOOS/$GOARCH |

api/run.go

Lines changed: 68 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,75 @@ import (
1212
"github.com/tetratelabs/func-e/internal/api"
1313
)
1414

15-
// HomeDir is an absolute path which most importantly contains "versions"
16-
// installed from EnvoyVersionsURL. Defaults to "${HOME}/.func-e"
15+
// Deprecated: Use ConfigHome, DataHome, StateHome or RuntimeDir instead.
16+
// This function will be removed in a future version.
1717
func HomeDir(homeDir string) RunOption {
1818
return func(o *api.RunOpts) {
19-
o.HomeDir = homeDir
19+
o.ConfigHome = homeDir
20+
o.DataHome = homeDir
21+
o.StateHome = homeDir
22+
o.RuntimeDir = homeDir
23+
}
24+
}
25+
26+
// ConfigHome is the directory containing configuration files.
27+
// Defaults to "~/.config/func-e"
28+
//
29+
// Files stored here:
30+
// - envoy-version (selected version preference)
31+
func ConfigHome(configHome string) RunOption {
32+
return func(o *api.RunOpts) {
33+
o.ConfigHome = configHome
34+
}
35+
}
36+
37+
// DataHome is the directory containing downloaded Envoy binaries.
38+
// Defaults to "~/.local/share/func-e"
39+
//
40+
// Files stored here:
41+
// - envoy-versions/{version}/bin/envoy (downloaded Envoy binaries)
42+
func DataHome(dataHome string) RunOption {
43+
return func(o *api.RunOpts) {
44+
o.DataHome = dataHome
45+
}
46+
}
47+
48+
// StateHome is the directory containing persistent state like run logs.
49+
// Defaults to "~/.local/state/func-e"
50+
//
51+
// Files stored here:
52+
// - envoy-runs/{runID}/stdout.log,stderr.log (per-run logs)
53+
// - envoy-runs/{runID}/config_dump.json (Envoy configuration snapshot)
54+
func StateHome(stateHome string) RunOption {
55+
return func(o *api.RunOpts) {
56+
o.StateHome = stateHome
57+
}
58+
}
59+
60+
// RuntimeDir is the directory containing ephemeral runtime files.
61+
// Defaults to "/tmp/func-e-${UID}"
62+
//
63+
// Files stored here:
64+
// - {runID}/admin-address.txt (Envoy admin API endpoint)
65+
//
66+
// Note: Runtime files are ephemeral and may be cleaned up on system restart.
67+
func RuntimeDir(runtimeDir string) RunOption {
68+
return func(o *api.RunOpts) {
69+
o.RuntimeDir = runtimeDir
70+
}
71+
}
72+
73+
// RunID sets a custom run identifier used in StateDir and RuntimeDir paths.
74+
// By default, a timestamp-based runID is auto-generated (e.g., "20250115_123456_789").
75+
//
76+
// Use this to:
77+
// - Create predictable directories for Docker/K8s (e.g., RunID("0"))
78+
// - Implement custom naming schemes
79+
//
80+
// Validation: runID cannot contain path separators (/ or \)
81+
func RunID(runID string) RunOption {
82+
return func(o *api.RunOpts) {
83+
o.RunID = runID
2084
}
2185
}
2286

@@ -29,7 +93,7 @@ func EnvoyVersionsURL(envoyVersionsURL string) RunOption {
2993
}
3094

3195
// EnvoyVersion overrides the version of Envoy to run. Defaults to the
32-
// contents of "$HomeDir/versions/version".
96+
// contents of "$ConfigHome/envoy-version".
3397
//
3498
// When that file is missing, it is generated from ".latestVersion" from the
3599
// EnvoyVersionsURL. Its value can be in full version major.minor.patch format,

e2e/README.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,15 @@ go test -parallel 1 -v -failfast ./e2e
1616
When run via `make`, `func-e` is built on-demand by `$(current_binary)` target (same as `make build`).
1717
Ex. `$PWD/build/func-e_darwin_arm64/func-e`
1818

19-
It is also a good idea to override `FUNC_E_HOME` when running e2e, since by default it uses `$HOME/.func-e`.
19+
It is also a good idea to override the data directories when running e2e to avoid interfering with your local environment.
20+
The defaults are `~/.local/share/func-e`, `~/.local/state/func-e` and `/tmp/func-e-${UID}`.
2021

2122
```bash
22-
FUNC_E_HOME=/tmp/test make e2e
23+
FUNC_E_DATA_HOME=/tmp/test FUNC_E_STATE_HOME=/tmp/test FUNC_E_RUNTIME_DIR=/tmp/test make e2e
2324
```
2425

26+
Note: The deprecated `FUNC_E_HOME` environment variable is still supported for backwards compatibility.
27+
2528
## Envoy version list
2629
If the `func-e` version is a snapshot and "envoy-versions.json" exists, tests run against the local. This allows local
2730
development and pull requests to verify changes not yet [published](https://archive.tetratelabs.io/envoy/envoy-versions.json)

e2e/func-e_run_test.go

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,8 +21,12 @@ func TestRun_LogWarn(t *testing.T) {
2121
e2e.TestRun_LogWarn(t.Context(), t, funcEFactory{})
2222
}
2323

24-
func TestRun_RunDirectory(t *testing.T) {
25-
e2e.TestRun_RunDirectory(t.Context(), t, funcEFactory{})
24+
func TestRun_RunDir(t *testing.T) {
25+
// For binary e2e tests, state directory is controlled via FUNC_E_STATE_HOME env var
26+
// (CLI layer), not library API options like api.StateHome().
27+
stateDir := t.TempDir()
28+
t.Setenv("FUNC_E_STATE_HOME", stateDir)
29+
e2e.TestRun_RunDir(t.Context(), t, funcEFactory{}, stateDir)
2630
}
2731

2832
func TestRun_InvalidConfig(t *testing.T) {
@@ -36,3 +40,7 @@ func TestRun_StaticFile(t *testing.T) {
3640
func TestRun_CtrlCs(t *testing.T) {
3741
e2e.TestRun_CtrlCs(t.Context(), t, funcEFactory{})
3842
}
43+
44+
func TestRun_LegacyHomeDir(t *testing.T) {
45+
e2e.TestRun_LegacyHomeDir(t.Context(), t, funcEFactory{})
46+
}

e2e/func-e_test.go

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,9 @@ import (
1717
"time"
1818

1919
"github.com/tetratelabs/func-e/experimental/admin"
20+
internaladmin "github.com/tetratelabs/func-e/internal/admin"
2021
"github.com/tetratelabs/func-e/internal/test/build"
21-
"github.com/tetratelabs/func-e/internal/test/e2e"
22+
internale2e "github.com/tetratelabs/func-e/internal/test/e2e"
2223
)
2324

2425
var (
@@ -72,17 +73,24 @@ func funcEExec(ctx context.Context, args ...string) (string, string, error) {
7273
return stdout.String(), stderr.String(), err
7374
}
7475

75-
// funcEFactory implements runtest.FuncEFactory for E2E tests using a compiled func-e binary.
76+
// funcEFactory implements internale2e.FuncEFactory for E2E tests using a
77+
// compiled func-e binary.
7678
type funcEFactory struct{}
7779

78-
func (funcEFactory) New(_ context.Context, _ *testing.T, stdout, stderr io.Writer) (e2e.FuncE, error) {
80+
func (funcEFactory) New(_ context.Context, _ *testing.T, stdout, stderr io.Writer) (internale2e.FuncE, error) {
7981
return &funcE{stdout: stdout, stderr: stderr}, nil
8082
}
8183

82-
// funcE implements runtest.FuncE for e2e tests using the compiled binary
84+
// funcE implements internale2e.FuncE for e2e tests using the compiled binary
8385
type funcE struct {
8486
cmd *exec.Cmd
8587
stdout, stderr io.Writer
88+
envoyPid int
89+
}
90+
91+
// EnvoyPid implements the same method as documented on internale2e.FuncE
92+
func (a *funcE) EnvoyPid() int {
93+
return a.envoyPid
8694
}
8795

8896
// OnStart inspects the running func-e process tree to find the Envoy process and its run directory,
@@ -93,7 +101,14 @@ func (a *funcE) OnStart(ctx context.Context) (admin.AdminClient, error) {
93101
}
94102
funcEPid := a.cmd.Process.Pid
95103

96-
adminClient, err := admin.NewAdminClient(ctx, funcEPid)
104+
// Poll for the admin address path from the Envoy process command line
105+
envoyPid, adminAddressPath, err := internaladmin.PollEnvoyPidAndAdminAddressPath(ctx, funcEPid)
106+
if err != nil {
107+
return nil, err
108+
}
109+
a.envoyPid = envoyPid
110+
111+
adminClient, err := internaladmin.NewAdminClient(ctx, adminAddressPath)
97112
if err == nil {
98113
err = adminClient.AwaitReady(ctx, 100*time.Millisecond)
99114
}

0 commit comments

Comments
 (0)