Cache Policy - POC

[danielnginx]

• Determine spec of cache policy
• Design Document
• Defaults: Nginx defaults will be used
• Minimum configuration
• Investigate: Exclude Response Code and Methods => Is that possible? Possible but with caveat of no persistence for specified codes (implemented using 0s cache persistence)

Resources:

https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/

[vepatel]

Configmap:

    entries:
      http-snippets: |
        proxy_cache_path /etc/nginx/cache keys_zone=mycache:10m;
        geo $purge_allowed {
           default         0;  # deny
            <ip_addr>   1;  # allow
       }

        map $request_method $purge_method {
           PURGE   $purge_allowed;
           default 0;
       }

VS:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
spec:
  server-snippets: |
    proxy_cache mycache;
    proxy_cache_valid 200 10m;
    proxy_cache_valid 201 0s;
    proxy_cache_methods GET;
    if ($request_method = POST) {
      set $no_cache 1;
    }
    proxy_no_cache $no_cache;
    proxy_ignore_headers Cache-Control Expires;
    add_header X-Cache-Status $upstream_cache_status;
  host: cafe.example.com
  tls:
    secret: cafe-secret
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  - name: coffee
    service: coffee-svc
    port: 80
  routes:
  - path: /tea
    action:
      pass: tea
  - path: /coffee
    action:
      pass: coffee
  - path: /water
    action:
      return:
        code: 201
        type: "text/plain"
        body: "Water is not cached"

cURL:

 curl -I -X GET --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/tea --insecure
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Wed, 14 May 2025 16:08:04 GMT
Content-Type: text/plain
Content-Length: 154
Connection: keep-alive
Expires: Wed, 14 May 2025 16:08:03 GMT
Cache-Control: no-cache
**X-Cache-Status: MISS**

curl -I -X GET --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/tea --insecure
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Wed, 14 May 2025 16:08:07 GMT
Content-Type: text/plain
Content-Length: 154
Connection: keep-alive
Expires: Wed, 14 May 2025 16:08:03 GMT
Cache-Control: no-cache
**X-Cache-Status: HIT**

curl -I -X **POST** --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/tea --insecure
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Wed, 14 May 2025 16:14:59 GMT
Content-Type: text/plain
Content-Length: 154
Connection: keep-alive  #POST method won't be cached this no EXPIRES etc.

curl -I -X GET --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/tea --insecure
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Wed, 14 May 2025 16:19:29 GMT
Content-Type: text/plain
Content-Length: 154
Connection: keep-alive
Expires: Wed, 14 May 2025 16:19:28 GMT
Cache-Control: no-cache
**X-Cache-Status: EXPIRED**


curl -I -X GET --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/water --insecure
HTTP/1.1 201 Created
Server: nginx/1.27.4
Date: Wed, 14 May 2025 16:21:04 GMT
Content-Type: text/plain
Content-Length: 19
Connection: keep-alive
**X-Cache-Status: MISS** #will always a MISS because 201 is set to 0s, this can be used for response code exclusion, see request below

curl -I -X GET --resolve cafe.example.com:$IC_HTTPS_PORT:$IC_IP https://cafe.example.com:$IC_HTTPS_PORT/water --insecure
HTTP/1.1 201 Created
Server: nginx/1.27.4
Date: Wed, 14 May 2025 16:28:10 GMT
Content-Type: text/plain
Content-Length: 19
Connection: keep-alive
**X-Cache-Status: MISS**

nginx.conf:

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    map_hash_max_size 2048;
    map_hash_bucket_size 256;


    js_import /etc/nginx/njs/apikey_auth.js;
    js_set $apikey_auth_hash apikey_auth.hash;
    
    proxy_cache_path /etc/nginx/cache keys_zone=mycache:10m;
    

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

vs conf:

server {
    listen 80;
    listen [::]:80;


    server_name cafe.example.com;

    set $resource_type "virtualserver";
    set $resource_name "cafe";
    set $resource_namespace "default";
    listen 443 ssl;
    listen [::]:443 ssl;

    ssl_certificate $secret_dir_path/default-cafe-secret;
    ssl_certificate_key $secret_dir_path/default-cafe-secret;

    server_tokens "on";
    proxy_cache mycache;
    proxy_cache_valid 200 10m;
    proxy_cache_valid 201 0s;
    proxy_cache_methods GET;
    if ($request_method = POST) {
      set $no_cache 1;
    }
    proxy_no_cache $no_cache;
    proxy_ignore_headers Cache-Control Expires;
    add_header X-Cache-Status $upstream_cache_status;
    

cache dir contents:

nginx@test-release-nginx-ingress-controller-747bbbc87f-nf9zd:/$ ls  /etc/nginx/cache
094962651ff7871d707e81819503b884  25acf017febf6f91bb6eea6f9fbc783e  c5e12aa5a0d94bc989d9ac0e3abaf437

nginx@test-release-nginx-ingress-controller-747bbbc87f-nf9zd:/$ cat /etc/nginx/cache/c5e12aa5a0d94bc989d9ac0e3abaf437
�N,h��������}L,h4�
                  �{L
KEY: http://vs_default_cafe_coffee/coffee
HTTP/1.1 200 OK
Server: nginx/1.27.5
Date: Tue, 20 May 2025 09:33:49 GMT
Content-Type: text/plain
Content-Length: 160
Connection: close
Expires: Tue, 20 May 2025 09:33:48 GMT
Cache-Control: no-cache

Server address: 10.40.1.10:8080
Server name: coffee-676c9f8944-lrtbk
Date: 20/May/2025:09:33:49 +0000
URI: /coffee
Request ID: b5bad807b81cf371d05f9c29a2b1339c

nginx@test-release-nginx-ingress-controller-747bbbc87f-nf9zd:/$ cat /etc/nginx/cache/25acf017febf6f91bb6eea6f9fbc783e
�O,h���������M,h��qwH
KEY: http://vs_default_webapp_webapp/
HTTP/1.1 200 OK
Server: nginx/1.27.5
Date: Tue, 20 May 2025 09:38:09 GMT
Content-Type: text/plain
Content-Length: 154
Connection: close
Expires: Tue, 20 May 2025 09:38:08 GMT
Cache-Control: no-cache

Server address: 10.40.0.25:8080
Server name: webapp-7b7dfbff54-w5w7q
Date: 20/May/2025:09:38:09 +0000
URI: /
Request ID: d2b2bdfc9f27fa00c4ad0ff0437c0d8b

nginx@test-release-nginx-ingress-controller-747bbbc87f-nf9zd:/$ cat /etc/nginx/cache/094962651ff7871d707e81819503b884
�M,h���������K,h�Q��uF
KEY: http://vs_default_cafe_tea/tea
HTTP/1.1 200 OK
Server: nginx/1.27.5
Date: Tue, 20 May 2025 09:30:06 GMT
Content-Type: text/plain
Content-Length: 154
Connection: close
Expires: Tue, 20 May 2025 09:30:05 GMT
Cache-Control: no-cache

Server address: 10.40.0.24:8080
Server name: tea-6fbfdcb95d-rt94k
Date: 20/May/2025:09:30:06 +0000
URI: /tea
Request ID: 80b5cb2eb61a44afe764857ecc346eb3
nginx@test-release-nginx-ingress-controller-747bbbc87f-nf9zd:/$ 

***For testing proxy_ignore_headers Cache-Control Expires; is used in snippets as upstream cafe app is setting Cache-Control: no-cache header resulting in no cache, we can also configure this in VS using action.proxy.responseHeaders as seen below:

  upstreams:
  - name: webapp
    service: webapp-svc
    port: 80
  routes:
  - path: /
    action:
      proxy:
        upstream: webapp
        responseHeaders:
           ignore:
            - Expires
            - Cache-Control

[vepatel]

Proposed design: Option 1: REJECTED

Configmap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
data:
  cache-path: "/etc/nginx/cache" #Required
  cache-zone-name: "mycache" #Required
  cache-zone-size: "10m" #Required
  cache-purge-allow: ["10.0.0.1","127.0.0.1"] #optional

Policy:

apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
  name: my-cache-policy
spec:
  cache:
    cacheZone: "mycache" #Required
    allowedCodes: [200, 301, 302] #Optional
    allowedMethods: ["GET", "POST"] #Optional
    time: "10m" #Optional

Validation considerations:

• zone-name must match between configmap and Policy i.e. proxy-cache-zone-name and cacheZone
• allowedCodes, allowedMethods and time should not be allowed without cacheZone (should fail at the kubectl apply)
• Only using cacheZone in policy results in nginx defaults i.e. caching will be enabled based on upstream headers
• User can only define one cache_zone per IC
• cache-purge-ips should only be allowed if its a N+ installation

Advantage: Define one zone, use multiple times in n number of policies

[vepatel]

Proposed design: Option 2: ACCEPTED

Configmap: No changes required.

Policy:

apiVersion: k8s.nginx.org/v1
kind: Policy
metadata:
  name: my-cache-policy
spec:
  cache:
    cacheZoneName: "mycache" #Required
    cacheZoneSize: "10m" #Required
    allowedCodes: [200, 301, 302] #Optional
    allowedMethods: ["GET", "POST"] #Optional
    time: "10m" #Optional
    cachePurgeAllow: ["10.0.0.1","127.0.0.0/24"] #Optional
    overrideUpstreamCache: "true" #boolean optional, false by default
    levels: "1:2" #directory levels

Validation considerations:

• cacheZoneName and cacheZoneSize are required fields and at minimum all 3 should be in policy
• allowedCodes, allowedMethods and time should not be allowed without 3 keys above (should fail at the kubectl apply)
• only using cacheZoneName and cacheZoneSize in policy results in nginx defaults i.e. caching will be enabled based on upstream headers
• cacheZoneName should always be unique across n policies (n policies mean n zones)
• cachePurgeAllow should only be allowed if this is a N+ installation.

  Warning  AddedOrUpdatedWithError  3m3s  nginx-ingress-controller  Configuration for default/cafe was added or updated ; but was not applied: error reloading NGINX for VirtualServer default/cafe: nginx reload failed: command /usr/sbin/nginx -s reload -e stderr stdout: ""
stderr: "2025/05/20 15:52:21 [emerg] 171#171: the same path name \"/etc/nginx/cache\" used in /etc/nginx/conf.d/vs_default_cafe.conf:13 and in /etc/nginx/conf.d/vs_default_webapp.conf:7\n"
finished with error: exit status 1
  Warning  AddedOrUpdatedWithError  2m17s  nginx-ingress-controller  Configuration for default/cafe was added or updated ; but was not applied: error reloading NGINX for VirtualServer default/cafe: nginx reload failed: command /usr/sbin/nginx -s reload -e stderr stdout: ""
stderr: "2025/05/20 15:53:07 [emerg] 173#173: duplicate zone \"mycache\" in /etc/nginx/conf.d/vs_default_webapp.conf:7\n"

Advantage: Can have multiple paths and zones per host i.e. per VS

VS example with snippets:

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: cafe
spec:
  http-snippets: |
    proxy_cache_path /etc/nginx/cache2 keys_zone=mycache2:10m;
    geo $purge_allowed {
           default         0;  # deny
           <ip_addr>  1;  # allow
     }

    map $request_method $purge_method {
           PURGE   $purge_allowed;
           default 0;
    }
  server-snippets: |
    proxy_cache mycache2;
    proxy_cache_valid 200 10m;
    proxy_cache_methods GET;
    proxy_ignore_headers Cache-Control Expires;
    add_header X-Cache-Status $upstream_cache_status;
  host: cafe.example.com
  tls:
    secret: cafe-secret
  upstreams:
  - name: tea
    service: tea-svc
    port: 80
  - name: coffee
    service: coffee-svc
    port: 80
  routes:
  - path: /tea
    action:
      pass: tea
  - path: /coffee
    action:
      pass: coffee

[vepatel]

• Purge functionality is only available with N+ https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_purge and will only be enabled if IC is running N+ and cachePurgeAllow is non empty list

 Configuration for default/cafe was added or updated ; but was not applied: error reloading NGINX for VirtualServer default/cafe: nginx reload failed: command /usr/sbin/nginx -s reload -e stderr stdout: ""
stderr: "2025/05/21 15:08:37 [emerg] 190#190: unknown directive \"proxy_cache_purge\" in /etc/nginx/conf.d/vs_default_cafe.conf:41\n"

• cacheZoneName should be used as predefined path suffix
  eg: cacheZoneName: "myzone1" will result in absolute path/etc/nginx/myzone1.
  readonlyrootfilesystem will cause errors if user tried to create multi level directories unless we create one via initContainer.
• consider having a dedicated path for content caching so as not to pollute /etc/nginx, /var/cache/nginx is a good option
• Minimum config should only set proxy_cache_path in http and proxy_cache in server or location context
• Policy can be applied at both spec, route and subroute(VSR) level.
• Policy merging behaviour: A VS/VSR can reference only a single cache policy. Every subsequent reference will be ignored.
• cacheZoneName should be used as predefined path suffix combined with parent VS or VSR name