Global limit do not support the HTTP protocol but support HTTPS, my redis uses the Sentinel mode

[wqzgithub]

Description: Global limit do not support the HTTP protocol but support HTTPS

What issue is being seen? Describe what should be happening instead of
the bug, for example: The expected value isn't returned, etc.

Repro steps:

Include sample requests, environment, etc. All data and inputs
required to reproduce the bug.
add BackendTrafficPolicy，from https://gateway.envoyproxy.io/docs/tasks/traffic/global-rate-limit/

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Environment: version 1.5.0

Include the environment like gateway version, envoy version and so on.
envoygateway 1.5.0 , my environment used operater deployed
Logs:
Include the access logs and the Envoy logs.
ratelimit notify success load config...
envoy proxy has no error logs

[wqzgithub]

The yaml I provided is from a public network verification environment rather than a production one. There might be some Refs errors, but this does not affect my problem feedback.

[zirain]

the hostname in the httproute didn't match the one in the Gateway (listener), are you sure that the httproute was accepted?

[wqzgithub]

yes, I sure ,
This is a random copy of the yaml in my internal test environment. However, it is definitely working fine in my internal network production. My https can limit the rate, but http cannot. Both protocols can access my backend normally. I will provide the produced yaml

[wqzgithub]

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: psc2-t
  namespace: envoy-gateway-space
spec:
  hostnames:
    - gateway.myapp.cn
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: region-zk-test-gateway
      namespace: envoy-gateway-space
  rules:
    - backendRefs:
        - group: gateway.envoyproxy.io
          kind: Backend
          name: psc2-t
          namespace: envoy-gateway-space
          weight: 1
      matches:
        - headers:
            - name: appId
              type: RegularExpression
              value: ^.+$
          path:
            type: PathPrefix
            value: /psc-process/

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: psc2-t
  namespace: envoy-gateway-space
spec:
  endpoints:
    - fqdn:
        hostname: psc2-t.example.cn
        port: 80
  type: Endpoints

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: limit-psc2-t
  namespace: envoy-gateway-space
spec:
  rateLimit:
    global:
      rules:
        - clientSelectors:
            - headers:
              - invert: false
                name: appId
                type: Exact
                value: CSP
          limit:
            requests: 3
            unit: Minute
    type: Global
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: psc2-t
      namespace: envoy-gateway-space

[zirain]

I cannot reprod with following:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      hostname: www.example.com
    - name: https
      protocol: HTTPS
      port: 443
      hostname: www.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: example-cert
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-ratelimit
spec:
  parentRefs:
    - name: eg
  hostnames:
    - www.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: policy-httproute
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: http-ratelimit
  rateLimit:
    type: Global
    global:
      rules:
        - clientSelectors:
            - headers:
                - name: x-user-id
                  value: two
          limit:
            requests: 3
            unit: Minute

[wqzgithub]

Could you provide your envoy proxy configmap, which is the configuration related to redis? I will check my configuration based on this configuration. Thank you!

[wqzgithub]

I tried to switch to single-point redis, but the problem still exists. I don't think it's a problem with redis itself. Here is my gateway configuration

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: region-zk-test-gateway
  namespace: envoy-gateway-space
spec:
  addresses:
    - type: Hostname
      value: gateway.myapp.cn
    - type: IPAddress
      value: xx.xxx.xxx.14
    - type: IPAddress
      value: xxxx:xxxx:0:xx::1:14
  gatewayClassName: envoy-gateway-operator-cpaas-default
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: envoy-proxy
  listeners:
    - allowedRoutes:
        namespaces:
          from: All
      hostname: gateway.myapp.cn
      name: http
      port: 80
      protocol: HTTP
    - allowedRoutes:
        namespaces:
          from: All
      hostname: gateway.myapp.cn
      name: https
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: ueapi-t-tls-cert
        mode: Terminate

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
extensionApis:
  enableBackend: true
gateway:
  controllerName: gatewayapi.cpaas.io/envoy-gateway-operator-cpaas-default
logging:
  level:
    default: info
provider:
  kubernetes:
    deploy:
      type: ControllerNamespace
    rateLimitDeployment:
      container:
        env:
        - name: REDIS_AUTH
          value: passwd
        image: registry/envoyproxy/ratelimit:a90e0e5d
        resources:
          limits:
            cpu: 1
            memory: 1024Mi
          requests:
            cpu: 100m
            memory: 256Mi
      patch:
        type: StrategicMerge
        value:
          spec:
            template:
              spec:
                containers:
                - imagePullPolicy: IfNotPresent
                  name: envoy-ratelimit
      replicas: 2
    shutdownManager:
      image: registry/envoyproxy/gateway:v1.5.0
  type: Kubernetes
rateLimit:
  backend:
    redis:
      url: redis://alone-redis-ip:6379
    type: Redis

[zirain]

This's something related to your configuration.
Please check your the diff between your configuration with I provided above.

[wqzgithub]

The issue is that when the gateway has multiple listeners, rate limiting can only take effect on one. How can I ensure that all listeners take effect
@zirain

[zirain]

when the gateway has multiple listeners, rate limiting can only take effect on one
I don't think that's true.

- address:
    socketAddress:
      address: 0.0.0.0
      portValue: 19003
  bypassOverloadManager: true
  filterChains:
  - filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        httpFilters:
        - name: envoy.filters.http.health_check
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.health_check.v3.HealthCheck
            headers:
            - name: :path
              stringMatch:
                exact: /ready
            passThroughMode: false
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            suppressEnvoyHeaders: true
        routeConfig:
          name: ready_route
          virtualHosts:
          - domains:
            - '*'
            name: ready_route
            routes:
            - directResponse:
                status: 500
              match:
                prefix: /
        statPrefix: eg-ready-http
  name: envoy-gateway-proxy-ready-0.0.0.0-19003
- accessLog:
  - filter:
      responseFlagFilter:
        flags:
        - NR
    name: envoy.access_loggers.file
    typedConfig:
      '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      logFormat:
        jsonFormat:
          :authority: '%REQ(:AUTHORITY)%'
          bytes_received: '%BYTES_RECEIVED%'
          bytes_sent: '%BYTES_SENT%'
          connection_termination_details: '%CONNECTION_TERMINATION_DETAILS%'
          downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
          downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
          duration: '%DURATION%'
          method: '%REQ(:METHOD)%'
          protocol: '%PROTOCOL%'
          requested_server_name: '%REQUESTED_SERVER_NAME%'
          response_code: '%RESPONSE_CODE%'
          response_code_details: '%RESPONSE_CODE_DETAILS%'
          response_flags: '%RESPONSE_FLAGS%'
          route_name: '%ROUTE_NAME%'
          start_time: '%START_TIME%'
          upstream_cluster: '%UPSTREAM_CLUSTER%'
          upstream_host: '%UPSTREAM_HOST%'
          upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
          upstream_transport_failure_reason: '%UPSTREAM_TRANSPORT_FAILURE_REASON%'
          user-agent: '%REQ(USER-AGENT)%'
          x-envoy-origin-path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
          x-envoy-upstream-service-time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
          x-forwarded-for: '%REQ(X-FORWARDED-FOR)%'
          x-request-id: '%REQ(X-REQUEST-ID)%'
      path: /dev/stdout
  address:
    socketAddress:
      address: 0.0.0.0
      portValue: 10080
  defaultFilterChain:
    filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        accessLog:
        - name: envoy.access_loggers.file
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            logFormat:
              jsonFormat:
                :authority: '%REQ(:AUTHORITY)%'
                bytes_received: '%BYTES_RECEIVED%'
                bytes_sent: '%BYTES_SENT%'
                connection_termination_details: '%CONNECTION_TERMINATION_DETAILS%'
                downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
                downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
                duration: '%DURATION%'
                method: '%REQ(:METHOD)%'
                protocol: '%PROTOCOL%'
                requested_server_name: '%REQUESTED_SERVER_NAME%'
                response_code: '%RESPONSE_CODE%'
                response_code_details: '%RESPONSE_CODE_DETAILS%'
                response_flags: '%RESPONSE_FLAGS%'
                route_name: '%ROUTE_NAME%'
                start_time: '%START_TIME%'
                upstream_cluster: '%UPSTREAM_CLUSTER%'
                upstream_host: '%UPSTREAM_HOST%'
                upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
                upstream_transport_failure_reason: '%UPSTREAM_TRANSPORT_FAILURE_REASON%'
                user-agent: '%REQ(USER-AGENT)%'
                x-envoy-origin-path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
                x-envoy-upstream-service-time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
                x-forwarded-for: '%REQ(X-FORWARDED-FOR)%'
                x-request-id: '%REQ(X-REQUEST-ID)%'
            path: /dev/stdout
        commonHttpProtocolOptions:
          headersWithUnderscoresAction: REJECT_REQUEST
        http2ProtocolOptions:
          initialConnectionWindowSize: 1048576
          initialStreamWindowSize: 65536
          maxConcurrentStreams: 100
        httpFilters:
        - name: envoy.filters.http.ratelimit
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            disableXEnvoyRatelimitedHeader: true
            domain: default/eg/http
            enableXRatelimitHeaders: DRAFT_VERSION_03
            rateLimitService:
              grpcService:
                envoyGrpc:
                  clusterName: ratelimit_cluster
              transportApiVersion: V3
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            suppressEnvoyHeaders: true
        mergeSlashes: true
        normalizePath: true
        pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
        rds:
          configSource:
            ads: {}
            resourceApiVersion: V3
          routeConfigName: default/eg/http
        serverHeaderTransformation: PASS_THROUGH
        statPrefix: http-10080
        useRemoteAddress: true
    name: default/eg/http
  maxConnectionsToAcceptPerSocketEvent: 1
  name: default/eg/http
  perConnectionBufferLimitBytes: 32768
- accessLog:
  - filter:
      responseFlagFilter:
        flags:
        - NR
    name: envoy.access_loggers.file
    typedConfig:
      '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      logFormat:
        jsonFormat:
          :authority: '%REQ(:AUTHORITY)%'
          bytes_received: '%BYTES_RECEIVED%'
          bytes_sent: '%BYTES_SENT%'
          connection_termination_details: '%CONNECTION_TERMINATION_DETAILS%'
          downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
          downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
          duration: '%DURATION%'
          method: '%REQ(:METHOD)%'
          protocol: '%PROTOCOL%'
          requested_server_name: '%REQUESTED_SERVER_NAME%'
          response_code: '%RESPONSE_CODE%'
          response_code_details: '%RESPONSE_CODE_DETAILS%'
          response_flags: '%RESPONSE_FLAGS%'
          route_name: '%ROUTE_NAME%'
          start_time: '%START_TIME%'
          upstream_cluster: '%UPSTREAM_CLUSTER%'
          upstream_host: '%UPSTREAM_HOST%'
          upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
          upstream_transport_failure_reason: '%UPSTREAM_TRANSPORT_FAILURE_REASON%'
          user-agent: '%REQ(USER-AGENT)%'
          x-envoy-origin-path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
          x-envoy-upstream-service-time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
          x-forwarded-for: '%REQ(X-FORWARDED-FOR)%'
          x-request-id: '%REQ(X-REQUEST-ID)%'
      path: /dev/stdout
  address:
    socketAddress:
      address: 0.0.0.0
      portValue: 10443
  filterChains:
  - filterChainMatch:
      serverNames:
      - www.example.com
    filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        accessLog:
        - name: envoy.access_loggers.file
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            logFormat:
              jsonFormat:
                :authority: '%REQ(:AUTHORITY)%'
                bytes_received: '%BYTES_RECEIVED%'
                bytes_sent: '%BYTES_SENT%'
                connection_termination_details: '%CONNECTION_TERMINATION_DETAILS%'
                downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
                downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
                duration: '%DURATION%'
                method: '%REQ(:METHOD)%'
                protocol: '%PROTOCOL%'
                requested_server_name: '%REQUESTED_SERVER_NAME%'
                response_code: '%RESPONSE_CODE%'
                response_code_details: '%RESPONSE_CODE_DETAILS%'
                response_flags: '%RESPONSE_FLAGS%'
                route_name: '%ROUTE_NAME%'
                start_time: '%START_TIME%'
                upstream_cluster: '%UPSTREAM_CLUSTER%'
                upstream_host: '%UPSTREAM_HOST%'
                upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
                upstream_transport_failure_reason: '%UPSTREAM_TRANSPORT_FAILURE_REASON%'
                user-agent: '%REQ(USER-AGENT)%'
                x-envoy-origin-path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
                x-envoy-upstream-service-time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
                x-forwarded-for: '%REQ(X-FORWARDED-FOR)%'
                x-request-id: '%REQ(X-REQUEST-ID)%'
            path: /dev/stdout
        commonHttpProtocolOptions:
          headersWithUnderscoresAction: REJECT_REQUEST
        http2ProtocolOptions:
          initialConnectionWindowSize: 1048576
          initialStreamWindowSize: 65536
          maxConcurrentStreams: 100
        httpFilters:
        - name: envoy.filters.http.ratelimit
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
            disableXEnvoyRatelimitedHeader: true
            domain: default/eg/https
            enableXRatelimitHeaders: DRAFT_VERSION_03
            rateLimitService:
              grpcService:
                envoyGrpc:
                  clusterName: ratelimit_cluster
              transportApiVersion: V3
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
            suppressEnvoyHeaders: true
        mergeSlashes: true
        normalizePath: true
        pathWithEscapedSlashesAction: UNESCAPE_AND_REDIRECT
        rds:
          configSource:
            ads: {}
            resourceApiVersion: V3
          routeConfigName: default/eg/https
        serverHeaderTransformation: PASS_THROUGH
        statPrefix: https-10443
        useRemoteAddress: true
    name: default/eg/https
    transportSocket:
      name: envoy.transport_sockets.tls
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
        commonTlsContext:
          alpnProtocols:
          - h2
          - http/1.1
          tlsCertificateSdsSecretConfigs:
          - name: default/example-cert
            sdsConfig:
              ads: {}
              resourceApiVersion: V3
        disableStatefulSessionResumption: true
        disableStatelessSessionResumption: true
  listenerFilters:
  - name: envoy.filters.listener.tls_inspector
    typedConfig:
      '@type': type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector
  maxConnectionsToAcceptPerSocketEvent: 1
  name: default/eg/https
  perConnectionBufferLimitBytes: 32768
- address:
    socketAddress:
      address: 0.0.0.0
      portValue: 19001
  bypassOverloadManager: true
  filterChains:
  - filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        httpFilters:
        - name: envoy.filters.http.router
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
        normalizePath: true
        routeConfig:
          name: local_route
          virtualHosts:
          - domains:
            - '*'
            name: prometheus_stats
            routes:
            - match:
                headers:
                - name: :method
                  stringMatch:
                    exact: GET
                path: /stats/prometheus
              route:
                cluster: prometheus_stats
        statPrefix: eg-stats-http
  name: envoy-gateway-proxy-stats-0.0.0.0-19001

[wqzgithub]

It seems to occur with a random probability. I have four listeners, and one of them will limit the rate and not work. If you can't reproduce this phenomenon on your side, then I don't have more problems on my side at present

[arkodg]

closing for now since it cant be reproed, feel free to reopen if you can