cannot configure phoenix with envoy gateway in k8s?

[Dericko681]

I have deployed envoy gateway into a local Kubernetes cluster. I have tested with my open AI compatible model backend and it works, curl requests to the model return a 200 Ok response. I have installed and configured arize phoenix using the guide here https://aigateway.envoyproxy.io/docs/capabilities/observability/tracing but at the testing phase where i run kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces" no traces are found. Have you done something similar to this pls share to step by step guide with me.
Please help me here
Thank you

[RogerHYang]

Hi @Dericko681,

Thanks for reaching out! I did some testing and found the most likely cause.

Most Likely Issue: Envoy Gateway Version or Configuration

AI Gateway v0.4.0 requires Envoy Gateway v1.5.0+ (the docs recommend v1.5.4). I tested with v1.6.0 and verified it works. Older versions or missing the -f envoy-gateway-values.yaml flag will prevent traces from working.

Quick diagnostic — run this command and share the result:

kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller | grep "inserting AI Gateway extproc"

If you see no output, the ext_proc filter isn't being configured (which generates traces).

The Fix

Reinstall Envoy Gateway with the correct version and the AI Gateway values file:

helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \
    --version v1.6.0 \
    --namespace envoy-gateway-system \
    --create-namespace \
    -f https://raw.githubusercontent.com/envoyproxy/ai-gateway/v0.4.0/manifests/envoy-gateway-values.yaml

Both are critical:

• Version v1.6.0 — Verified working (docs minimum is v1.5.0+)
• Values file — Configures the extension server that enables tracing

After reinstalling, restart your gateway pods:

kubectl rollout restart deployment -n envoy-gateway-system

---

Other Things to Check

If the version is correct, here are additional troubleshooting steps:

1. Verify Phoenix service name matches your config:

kubectl get svc -n envoy-ai-gateway-system | grep phoenix

The service name is <helm-release-name>-svc. If you ran helm install phoenix ..., it would be phoenix-svc. Make sure OTEL_EXPORTER_OTLP_ENDPOINT in your AI Gateway config uses the correct name with FQDN:

http://phoenix-svc.envoy-ai-gateway-system.svc.cluster.local:6006

2. Verify Phoenix is receiving requests:

kubectl logs -n envoy-ai-gateway-system deployment/phoenix --tail=50 | grep "POST /v1/traces"

If you see 200 OK responses here, traces are being received and the issue may be elsewhere.

3. Generate test traffic:

kubectl port-forward -n envoy-gateway-system svc/<your-gateway-service> 8080:80

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{"model": "gpt-3.5-turbo", "messages": [{"role": "user", "content": "Hello"}]}'

---

Quick Reference

Issue	Solution
EG version too old	Use EG v1.6.0 (verified) or v1.5.4+ with AI Gateway v0.4.0
Missing values file	Always include -f envoy-gateway-values.yaml when installing EG
Service name mismatch	Use FQDN: http://phoenix-svc.envoy-ai-gateway-system.svc.cluster.local:6006
OTEL protocol	Add OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf to extProc env vars

---

Could you share:

1. Which Envoy Gateway version you installed?
2. The output of the diagnostic command above?

That will help us confirm the fix!

[Dericko681]

Hi @RogerHYang Thank you for attending to this issue

I have modified my deployment to use your recommended versions but kubectl logs -n envoy-ai-gateway-system deployment/phoenix | grep "POST /v1/traces" still returns no traces.

Here is the output of the diagnostic commands so that we can debug this further

[RogerHYang]

Thanks for sharing the screenshots! I can see:

• ✅ EG v1.6.0 with values file
• ✅ AI Gateway v0.4.0 with correct extProc.extraEnvVars parameter
• ✅ API requests return 200 OK

But I notice the "inserting AI Gateway extproc" log never appeared in your screenshots — even after the EG upgrade. This is the key issue: the extension hooks aren't being triggered.

Diagnostic: Check if extension manager is configured

First, let's verify the extension manager was properly configured:

kubectl get configmap -n envoy-gateway-system envoy-gateway-config -o yaml | grep -A 20 extensionManager

You should see extensionManager with hooks and service.fqdn.hostname: ai-gateway-controller.... If this returns nothing or shows no hooks, the values file wasn't applied correctly.

The Fix

Since you upgraded EG (REVISION: 3) rather than fresh install, the extension manager config may not have taken effect. Try:

Option A: Uninstall and reinstall EG with values file

helm uninstall eg -n envoy-gateway-system
helm install eg oci://docker.io/envoyproxy/gateway-helm \
    --version v1.6.0 \
    --namespace envoy-gateway-system \
    --create-namespace \
    -f https://raw.githubusercontent.com/envoyproxy/ai-gateway/v0.4.0/manifests/envoy-gateway-values.yaml

Option B: If you can't uninstall, restart everything in order

# 1. Restart AI Gateway controller first
kubectl rollout restart deployment -n envoy-ai-gateway-system ai-gateway-controller

# 2. Wait for it to be ready
kubectl rollout status deployment -n envoy-ai-gateway-system ai-gateway-controller

# 3. Delete Envoy pods to force recreation
kubectl delete pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic

# 4. Wait for new pods
kubectl wait --for=condition=Ready -n envoy-gateway-system \
    pods -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic --timeout=60s

Verify the fix

After restarting, check if the ext_proc filter is being inserted:

kubectl logs -n envoy-ai-gateway-system deployment/ai-gateway-controller --tail=50 | grep "inserting AI Gateway extproc"

If you see output like inserting AI Gateway extproc filter into listener, the fix worked.

Then verify OTEL env vars are in the sidecar:

ENVOY_POD=$(kubectl get pods -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-name=envoy-ai-gateway-basic -o jsonpath='{.items[0].metadata.name}')
kubectl get pod -n envoy-gateway-system $ENVOY_POD -o json | jq '.spec.initContainers[] | select(.name=="ai-gateway-extproc") | .env'

You should see OTEL_EXPORTER_OTLP_ENDPOINT in the output.

Let me know what you find!

[Dericko681]

Hi @RogerHYang here is the output of diagnostic commands

[Dericko681]

Hi @RogerHYang I have fixed it all. I guess the main problem in compatibility between ai gateway and envoy gateway versions

Thank you

[RogerHYang]

Awesome, glad you got it working! 🎉

[Dericko681]

AI Gateway v0.4.0 requires Envoy Gateway v1.5.0+ (the docs recommend v1.5.4). I tested with v1.6.0 and verified it works.