Different CEL calculation formulas are used across multiple models.

[Hyzhou]

Description:

I want to implement rate limiting based on usage costs in the Envoy AI Gateway.

I have two models with slightly different billing methods. I added a metadataKey to provide the billing metrics for the models.

This configuration does not achieve the desired effect. In the free model, the CEL seems to be using the CEL calculation formula of the other model.

Sample：

I accessed the free GLM45 model, but I noticed in the response headers that the "billing_charges" quota was decreasing.

apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: envoy-ai-gateway-tke-glm45
  namespace: default
spec:
  llmRequestCosts:
    - metadataKey: llm_input_token
      type: InputToken
    - metadataKey: llm_output_token
      type: OutputToken
    - metadataKey: llm_total_token
      type: TotalToken
    - metadataKey: billing_charges
      type: CEL
      cel: "0"
---
apiVersion: aigateway.envoyproxy.io/v1alpha1
kind: AIGatewayRoute
metadata:
  name: envoy-ai-gateway-tke-glm47
  namespace: default
spec:
  llmRequestCosts:
    - metadataKey: llm_input_token
      type: InputToken
    - metadataKey: llm_output_token
      type: OutputToken
    - metadataKey: llm_total_token
      type: TotalToken
    - metadataKey: billing_charges
      type: CEL
      cel: "int(double(input_tokens) / 1000.0 * 30.0 * 7.2 + double(output_tokens) / 1000.0 * 150.0 * 7.2)"
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"BackendTrafficPolicy","metadata":{"annotations":{},"name":"envoy-ai-gateway-default","namespace":"default"},"spec":{"rateLimit":{"global":{"rules":[{"clientSelectors":[{"headers":[{"name":"x-user-id","type":"Distinct"}]}],"cost":{"request":{"from":"Number","number":0},"response":{"from":"Metadata","metadata":{"key":"billing_charges","namespace":"io.envoy.ai_gateway"}}},"limit":{"requests":100000,"unit":"Day"}},{"cost":{"request":{"from":"Number","number":0},"response":{"from":"Metadata","metadata":{"key":"billing_charges","namespace":"io.envoy.ai_gateway"}}},"limit":{"requests":1000000,"unit":"Day"}}]},"type":"Global"},"targetRefs":[{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"envoy-ai-gateway-default"}]}}
  creationTimestamp: "2025-12-11T15:32:59Z"
  generation: 1
  name: envoy-ai-gateway-default
  namespace: default
  resourceVersion: "19951812"
  uid: b4986c34-cfe2-4758-afb0-10668815df71
spec:
  rateLimit:
    global:
      rules:
      - clientSelectors:
        - headers:
          - invert: false
            name: x-user-id
            type: Distinct
        cost:
          request:
            from: Number
            number: 0
          response:
            from: Metadata
            metadata:
              key: billing_charges
              namespace: io.envoy.ai_gateway
        limit:
          requests: 100000
          unit: Day
      - cost:
          request:
            from: Number
            number: 0
          response:
            from: Metadata
            metadata:
              key: billing_charges
              namespace: io.envoy.ai_gateway
        limit:
          requests: 1000000
          unit: Day
    type: Global

[Aditya7880900936]

Hi maintainers ,

This looks like a CEL scoping / reuse issue rather than a config problem.

Both routes define billing_charges using type: CEL, but with different formulas. When accessing the free model, it appears the CEL expression from the paid model is still being applied, which suggests CEL programs may be cached or bound globally instead of per-route.

I’d like to investigate how CEL expressions are compiled and scoped in the AI Gateway (route vs backend vs gateway level) and see whether the evaluator or cache key needs to include route identity.

Please let me know if it’s okay for me to start digging into this.

[aabchoo]

Hi @Hyzhou,

The filter config will set the LLMRequestCost for that specific metadata key once in gateway.go. An alternative solution could be to define a different metadata key for free vs paid. After, you can define a backend traffic policy for each.

@Aditya7880900936 feel free to do some more investigation on how CEL works. I am working on a proposal to set up token cost calculation on a per-backendRef level, so stay tuned for that!

[Aditya7880900936]

Thanks for the clarification, that makes sense 👍

I’ll take a closer look at how LLMRequestCost is initialized and scoped in gateway.go, and how CEL expressions are currently compiled and bound to metadata keys.

I agree that using different metadata keys works as a workaround today, but I’ll try to document the current behavior and see whether there’s a clean way to scope CEL evaluation more narrowly (e.g. per route or per backendRef), or at least make the limitation explicit in docs.

I’ll share findings once I’ve dug a bit deeper. Looking forward to the per-backendRef proposal as well.

[Aditya7880900936]

I dug into the controller code and confirmed the current behavior.

In reconcileFilterConfigSecret, LLMRequestCosts are aggregated across all attached AIGatewayRoutes and deduplicated purely by metadataKey. Once a key is seen, subsequent definitions (even with different CEL expressions) are skipped, and only the first one is written into the filter config.

This effectively makes LLMRequestCost gateway-scoped rather than route- or backend-scoped, which explains why the CEL formula from one model is applied to others sharing the same metadata key.

Using distinct metadata keys per model works as expected with the current design. Per-route or per-backendRef cost formulas would require a change in how costs are scoped and installed (which aligns with the per-backendRef proposal you mentioned).

[Hyzhou]

Hi @Hyzhou,

The filter config will set the LLMRequestCost for that specific metadata key once in gateway.go. An alternative solution could be to define a different metadata key for free vs paid. After, you can define a backend traffic policy for each.

@Aditya7880900936 feel free to do some more investigation on how CEL works. I am working on a proposal to set up token cost calculation on a per-backendRef level, so stay tuned for that!

Different models within the same BackendRef might also have different pricing strategies. I think this CEL expression should follow the definition of the AIGatewayRoute.
When we need to calculate the CEL, we should use the CEL rules defined on the corresponding AIGatewayRoute.

[Aditya7880900936]

That makes sense. Given the current gateway-scoped behavior, I agree this is more of a design limitation than a bug.

I think making the scoping rules explicit (or validating conflicting metadataKey usage) would help avoid confusion until per-backendRef or per-route cost calculation is supported.

Happy to help with docs or validation if that’s useful.

[yuzisun]

Hi @Hyzhou,
The filter config will set the LLMRequestCost for that specific metadata key once in gateway.go. An alternative solution could be to define a different metadata key for free vs paid. After, you can define a backend traffic policy for each.
@Aditya7880900936 feel free to do some more investigation on how CEL works. I am working on a proposal to set up token cost calculation on a per-backendRef level, so stay tuned for that!

Different models within the same BackendRef might also have different pricing strategies. I think this CEL expression should follow the definition of the AIGatewayRoute. When we need to calculate the CEL, we should use the CEL rules defined on the corresponding AIGatewayRoute.

This depends on how you setup the AIGatewayRoute, if your route is at the model level then same backendRef won’t have different pricing formulas, if your route is at the backend level then you need to create multiple routing rules per model so different model has its own rule and backendRef to define the pricing formulas.