Increasing Retry Attempts or Disabling Restart Behavior for OPA Bundle API Access

[daniel-pebble]

At present, OPA restarts after four unsuccessful attempts to access the bundle API. Is there a way to increase this number or disable this behavior entirely? We have OPA deployed as a sidecar, which receives data through the push method from a container running in the same pod. However, we would like to transition to a bundle API approach. If we modify the OPA injection configuration to enable the bundle API and the local container has not yet implemented the Bundle API, OPA will restart.

[anderseknert]

Hi @daniel-pebble 👋 Not sure I follow here... what do you mean by "OPA restarts"? AFAIK, OPA never shuts down other than when told so by a signal sent from the outside (SIGTERM, SIGKILL, etc). Applications can't really restart themselves, as once they shut down they're... well, shut down 🙂 I'm guessing you have some outside process polling OPA for readyness? And eventually decides that it has waited long enough. If you could share some more details, that would be helpful.

[daniel-pebble]

Hi @anderseknert , thanks for the prompt response. I did check the pod events and as far as I can tell k8s isn't restarting the container.
The readiness/liveness probe are getting back 500 errors.
I'm assuming the 500's are coming from OPA, is this because it can't contact the bundle API?

Readiness probe failed: Get "http://10.244.0.67:8282/health?plugins": dial tcp 10.244.0.67:8282: connect: connection refused
Readiness probe failed: Get "http://10.244.0.67:15021/healthz/ready": dial tcp 10.244.0.67:15021: connect: connection refused
Readiness probe failed: HTTP probe failed with statuscode: 500
Liveness probe failed: HTTP probe failed with statuscode: 500

{"addrs":["localhost:8181"],"diagnostic-addrs":["0.0.0.0:8282"],"level":"info","msg":"Initializing server.","time":"2023-03-31T19:23:34Z"}
{"level":"warning","msg":"OPA running with uid or gid 0. Running OPA with root privileges is not recommended.","time":"2023-03-31T19:23:34Z"}
{"level":"info","msg":"Starting bundle loader.","name":"pebble/oceans","plugin":"bundle","time":"2023-03-31T19:23:34Z"}
{"level":"info","msg":"Starting decision logger.","plugin":"decision_logs","time":"2023-03-31T19:23:34Z"}
{"addr":":9191","dry-run":false,"enable-reflection":false,"level":"info","msg":"Starting gRPC server.","path":"pebble/oceans/auth/allow","query":"","time":"2023-03-31T19:23:34Z"}
{"current_version":"0.49.0-envoy","download_opa":"https://openpolicyagent.org/downloads/v0.51.0/opa_linux_amd64","latest_version":"0.51.0","level":"info","msg":"OPA is out of date.","release_notes":"https://github.com/open-policy-agent/opa/releases/tag/v0.51.0","time":"2023-03-31T19:23:35Z"}
{"decision_id":"445d0474-38f2-4d67-8951-bf34124000a9","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"ec2d3191-7a73-4cfa-9fa2-25afe5c29afd"},"host":"10.244.0.67:8282","id":"18230573279788659512","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:23:36.459807Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":46970}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":600,"timer_rego_module_parse_ns":700,"timer_rego_query_compile_ns":74200,"timer_rego_query_eval_ns":140499,"timer_server_handler_ns":436697},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:23:36Z","timestamp":"2023-03-31T19:23:36.461449244Z","type":"openpolicyagent.org/decision_logs"}
{"client_addr":"127.0.0.6:51541","level":"info","msg":"Received request.","req_id":1,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:23:36Z"}
{"client_addr":"127.0.0.6:51541","level":"info","msg":"Sent response.","req_id":1,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.352297,"resp_status":500,"time":"2023-03-31T19:23:36Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get "http://52.252.243.18:9900/opa/bundle.tgz\": read tcp 10.244.0.67:57852-\u003e52.252.243.18:9900: read: connection reset by peer","name":"pebble/oceans","plugin":"bundle","time":"2023-03-31T19:23:37Z"}
{"decision_id":"6613afd0-d6e6-4983-b7ca-a2db75113345","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"562bf0ae-832c-463c-82b1-da3b2672c258"},"host":"10.244.0.67:8282","id":"4061562952638664502","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:23:44.439636Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":40834}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":400,"timer_rego_query_eval_ns":232599,"timer_server_handler_ns":444097},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:23:44Z","timestamp":"2023-03-31T19:23:44.440488821Z","type":"openpolicyagent.org/decision_logs"}
{"decision_id":"6045c233-3df0-4001-b6fd-8a54d60a4add","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"2e736480-baf3-4243-a96c-b4e6017faab8"},"host":"10.244.0.67:8282","id":"17524585752767896272","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:23:44.439308Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":40824}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":1200,"timer_rego_query_eval_ns":255998,"timer_server_handler_ns":523096},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:23:44Z","timestamp":"2023-03-31T19:23:44.44055802Z","type":"openpolicyagent.org/decision_logs"}
{"client_addr":"127.0.0.6:51541","level":"info","msg":"Received request.","req_id":2,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:23:44Z"}
{"client_addr":"127.0.0.6:46583","level":"info","msg":"Received request.","req_id":3,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:23:44Z"}
{"client_addr":"127.0.0.6:51541","level":"info","msg":"Sent response.","req_id":2,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.784494,"resp_status":500,"time":"2023-03-31T19:23:44Z"}
{"client_addr":"127.0.0.6:46583","level":"info","msg":"Sent response.","req_id":3,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.914094,"resp_status":500,"time":"2023-03-31T19:23:44Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get "http://52.252.243.18:9900/opa/bundle.tgz\": read tcp 10.244.0.67:57870-\u003e52.252.243.18:9900: read: connection reset by peer","name":"pebble/oceans","plugin":"bundle","time":"2023-03-31T19:23:44Z"}

{"decision_id":"6005562e-59b3-40fe-ae75-dbaaea8c4b72","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"751a12cc-ead2-4352-9e56-e34d7f90d25c"},"host":"10.244.0.67:8282","id":"14310980792228578821","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:23:54.438553Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":60354}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":700,"timer_rego_query_eval_ns":1035492,"timer_server_handler_ns":1303891},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:23:54Z","timestamp":"2023-03-31T19:23:54.440630954Z","type":"openpolicyagent.org/decision_logs"}
{"client_addr":"127.0.0.6:46583","level":"info","msg":"Received request.","req_id":4,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:23:54Z"}
{"client_addr":"127.0.0.6:46583","level":"info","msg":"Sent response.","req_id":4,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.325897,"resp_status":500,"time":"2023-03-31T19:23:54Z"}
{"decision_id":"64d45775-9daa-457e-881f-5d9e9b1ff9ee","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"bed35afa-6ea6-4023-82dd-8f2db512b733"},"host":"10.244.0.67:8282","id":"1393809065425954386","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:23:54.438624Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":60360}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":800,"timer_rego_query_eval_ns":4025670,"timer_server_handler_ns":4232369},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:23:54Z","timestamp":"2023-03-31T19:23:54.443728931Z","type":"openpolicyagent.org/decision_logs"}
{"client_addr":"127.0.0.6:41757","level":"info","msg":"Received request.","req_id":5,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:23:54Z"}
{"client_addr":"127.0.0.6:41757","level":"info","msg":"Sent response.","req_id":5,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.248198,"resp_status":500,"time":"2023-03-31T19:23:54Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get "http://52.252.243.18:9900/opa/bundle.tgz\": net/http: timeout awaiting response headers","name":"pebble/oceans","plugin":"bundle","time":"2023-03-31T19:23:54Z"}
{"decision_id":"1c75cada-2f8e-4630-b3e5-9389fb794a0f","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"a91b6e36-4199-4d58-9531-1cc45e01b041"},"host":"10.244.0.67:8282","id":"10838288731323879182","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:24:04.439362Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":60784}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":300,"timer_rego_query_eval_ns":1107292,"timer_server_handler_ns":1423490},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:24:04Z","timestamp":"2023-03-31T19:24:04.442058452Z","type":"openpolicyagent.org/decision_logs"}
{"decision_id":"07fa9b14-e04c-4064-bc3d-08f15150677d","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"d04c90ec-95ec-462e-b964-d944da46a921"},"host":"10.244.0.67:8282","id":"1007013781399409625","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:24:04.439631Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":60786}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":600,"timer_rego_query_eval_ns":1245891,"timer_server_handler_ns":1811887},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:24:04Z","timestamp":"2023-03-31T19:24:04.442113951Z","type":"openpolicyagent.org/decision_logs"}
{"client_addr":"127.0.0.6:42957","level":"info","msg":"Received request.","req_id":6,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:24:04Z"}
{"client_addr":"127.0.0.6:41757","level":"info","msg":"Received request.","req_id":7,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:24:04Z"}
{"client_addr":"127.0.0.6:41757","level":"info","msg":"Sent response.","req_id":7,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.646296,"resp_status":500,"time":"2023-03-31T19:24:04Z"}
{"client_addr":"127.0.0.6:42957","level":"info","msg":"Sent response.","req_id":6,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.254098,"resp_status":500,"time":"2023-03-31T19:24:04Z"}
{"decision_id":"5547bda0-4f2e-43ed-852f-74bb530918a8","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.67","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.0.67:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.26","x-forwarded-proto":"http","x-request-id":"a1e45ab7-f17e-4220-98b1-d928404bb16d"},"host":"10.244.0.67:8282","id":"14998172792787859047","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-03-31T19:24:04.449083Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.1","portValue":60798}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"605dd96e-0716-4693-9849-761679068a2c","version":"0.49.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":400,"timer_rego_query_eval_ns":143899,"timer_server_handler_ns":363797},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":true,"time":"2023-03-31T19:24:04Z","timestamp":"2023-03-31T19:24:04.450394791Z","type":"openpolicyagent.org/decision_logs"}
{"client_addr":"127.0.0.6:46583","level":"info","msg":"Received request.","req_id":8,"req_method":"GET","req_path":"/health","time":"2023-03-31T19:24:04Z"}
{"client_addr":"127.0.0.6:46583","level":"info","msg":"Sent response.","req_id":8,"req_method":"GET","req_path":"/health","resp_bytes":43,"resp_duration":0.401697,"resp_status":500,"time":"2023-03-31T19:24:04Z"}
{"level":"info","msg":"Shutting down...","time":"2023-03-31T19:24:04Z"}
{"level":"info","msg":"Server shutdown.","time":"2023-03-31T19:24:04Z"}
{"level":"info","msg":"Stopping bundle loader.","name":"pebble/oceans","plugin":"bundle","time":"2023-03-31T19:24:04Z"}
{"level":"error","msg":"Bundle load failed: request failed: Get "http://52.252.243.18:9900/opa/bundle.tgz\": context canceled","name":"pebble/oceans","plugin":"bundle","time":"2023-03-31T19:24:04Z"}
{"level":"info","msg":"Stopping decision logger.","plugin":"decision_logs","time":"2023-03-31T19:24:04Z"}

Pod config
name: istio-proxy
ports:
readinessProbe:
failureThreshold: 30
httpGet:
path: /healthz/ready
port: 15021
scheme: HTTP
initialDelaySeconds: 1
periodSeconds: 2
successThreshold: 1
timeoutSeconds: 3
...
name: opa-istio
livenessProbe:
failureThreshold: 3
httpGet:
path: /health?plugins
port: 8282
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /health?plugins
port: 8282
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
...
name: deploymentservice
livenessProbe:
failureThreshold: 3
httpGet:
path: /app-health/deploymentservice/livez
port: 15020
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /app-health/deploymentservice/readyz
port: 15020
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 1

OPA config

config.yaml: |
services:
- name: controller
url: http://52.252.243.18:9900
bundles:
pebble/oceans:
service: controller
resource: opa/bundle.tgz
plugins:
envoy_ext_authz_grpc:
addr: :9191
path: pebble/oceans/auth/allow
decision_logs:
console: true

[anderseknert]

Hi! Yeah, that's certainly Kubernetes killing your OPA instance. If you query the health endpoint yourself you'll likely see something like:

HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Date: Mon, 03 Apr 2023 06:36:06 GMT
Content-Length: 43

{"error":"one or more plugins are not up"}

If you want OPA to keep running regardless of the status of the plugins (like the bundle downloader), you can remove the ?plugins query parameter. Another, and perhaps better, option might be to raise the amount of time Kubernetes will attempt to ping the service before it kills it.

failureThreshold: 3

That number seems to correspond quite well to the "four unsucessful attempts" you mentioned.

[daniel-pebble]

Yeah that looks to be the cause. I think we'll wait until we roll out the bundle API to all backend services before switching to the OPA to use the bundle API. Seems like the safer route as disabling health checks doesn't feel right!
Thank you for your support @anderseknert