503 Service Unavailable - Using Bundle API #392

daniel-pebble · 2023-04-05T15:26:16Z

daniel-pebble
Apr 5, 2023

Hi,
I am currently attempting to transition from using external data via the push method to the bundle API. However, I've encountered an issue where OPA returns a 503 error after loading the bundle. To troubleshoot this, I have tried using a simplified bundle for testing purposes, but the same 503 error persists.

From what I can gather, the bundle appears to be loading correctly. I have enabled debug logs to investigate further, but I haven't identified any issues within the logs. However, I did notice that the result in the decision log displays as "null," which I presume is not a desirable outcome.

I would greatly appreciate any suggestions or guidance you may have to help resolve this issue.

##################
Env Details
##################
Kubernetes - v1.21.4
OPA - 0.51
Istio - 1.11.2

##################
curl http://10.244.1.49:8282/health?plugin -vvv

Trying 10.244.1.49:8282...
TCP_NODELAY set
Connected to 10.244.1.49 (10.244.1.49) port 8282 (#0)

GET /health?plugin HTTP/1.1
Host: 10.244.1.49:8282
User-Agent: curl/7.68.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< date: Wed, 05 Apr 2023 16:13:31 GMT
< server: istio-envoy
< x-envoy-decorator-operation: :0/*
< content-length: 0
<
Connection #0 to host 10.244.1.49 left intact

##################

kubectl logs deploymentservice-84d77d77c9-7kd85 opa-istio
{"addrs":["localhost:8181"],"diagnostic-addrs":["0.0.0.0:8282"],"level":"info","msg":"Initializing server.","time":"2023-04-05T16:13:25Z"}
{"level":"warning","msg":"OPA running with uid or gid 0. Running OPA with root privileges is not recommended.","time":"2023-04-05T16:13:25Z"}
{"level":"debug","msg":"maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined","time":"2023-04-05T16:13:25Z"}
{"level":"info","msg":"Starting bundle loader.","name":"pebble/oceans","plugin":"bundle","time":"2023-04-05T16:13:25Z"}
{"level":"info","msg":"Starting decision logger.","plugin":"decision_logs","time":"2023-04-05T16:13:25Z"}
{"level":"debug","msg":"Download starting.","time":"2023-04-05T16:13:25Z"}
{"headers":{"Prefer":["modes=snapshot,delta"],"User-Agent":["Open Policy Agent/0.51.0-envoy (linux, amd64)"]},"level":"debug","method":"GET","msg":"Sending request.","time":"2023-04-05T16:13:25Z","url":"http://localhost/opa/bundle"}
{"addr":":9191","dry-run":false,"enable-reflection":false,"level":"info","msg":"Starting gRPC server.","path":"pebble/oceans/auth/allow","query":"","time":"2023-04-05T16:13:25Z"}
{"headers":{"Content-Type":["application/json"],"User-Agent":["Open Policy Agent/0.51.0-envoy (linux, amd64)"]},"level":"debug","method":"POST","msg":"Sending request.","time":"2023-04-05T16:13:25Z","url":"https://telemetry.openpolicyagent.org/v1/version"}
{"level":"debug","msg":"Server initialized.","time":"2023-04-05T16:13:25Z"}
{"headers":{"Content-Disposition":["attachment; filename="bundle.tar.gz""],"Content-Type":["application/gzip"],"Date":["Wed, 05 Apr 2023 16:13:25 GMT"],"Etag":["05-BA-E5-10-DD-0A-2D-E9-1F-15-45-1A-CD-41-06-17"],"Server":["Kestrel"]},"level":"debug","method":"GET","msg":"Received response.","status":"200 OK","time":"2023-04-05T16:13:25Z","url":"http://localhost/opa/bundle"}
{"level":"debug","msg":"Download in progress.","time":"2023-04-05T16:13:25Z"}
{"level":"debug","msg":"Bundle activation in progress (). Opening storage transaction.","name":"pebble/oceans","plugin":"bundle","time":"2023-04-05T16:13:25Z"}
{"level":"debug","msg":"Opened storage transaction (4).","name":"pebble/oceans","plugin":"bundle","time":"2023-04-05T16:13:25Z"}
{"level":"debug","msg":"Closing storage transaction (4).","name":"pebble/oceans","plugin":"bundle","time":"2023-04-05T16:13:25Z"}
{"level":"info","msg":"Bundle loaded and activated successfully. Etag updated to 05-BA-E5-10-DD-0A-2D-E9-1F-15-45-1A-CD-41-06-17.","name":"pebble/oceans","plugin":"bundle","time":"2023-04-05T16:13:25Z"}
{"level":"debug","msg":"Waiting 1m36.51873759s before next download/retry.","time":"2023-04-05T16:13:25Z"}
{"headers":{"Content-Length":["216"],"Content-Type":["application/json"],"Date":["Wed, 05 Apr 2023 16:13:25 GMT"]},"level":"debug","method":"POST","msg":"Received response.","status":"200 OK","time":"2023-04-05T16:13:25Z","url":"https://telemetry.openpolicyagent.org/v1/version"}
{"current_version":"0.51.0-envoy","level":"debug","msg":"OPA is up to date.","time":"2023-04-05T16:13:25Z"}
{"decision-id":"7d453acb-0174-470c-8737-8defad28cd2e","level":"debug","msg":"no content-type header supplied, performing no body parsing","time":"2023-04-05T16:13:26Z"}
{"input":[[{"type":"string","value":"attributes"},{"type":"object","value":[[{"type":"string","value":"destination"},{"type":"object","value":[[{"type":"string","value":"address"},{"type":"object","value":[[{"type":"string","value":"socketAddress"},{"type":"object","value":[[{"type":"string","value":"address"},{"type":"string","value":"10.244.1.49"}],[{"type":"string","value":"portValue"},{"type":"number","value":8282}]]}]]}]]}],[{"type":"string","value":"metadataContext"},{"type":"object","value":[]}],[{"type":"string","value":"request"},{"type":"object","value":[[{"type":"string","value":"http"},{"type":"object","value":[[{"type":"string","value":"headers"},{"type":"object","value":[[{"type":"string","value":":authority"},{"type":"string","value":"10.244.1.49:8282"}],[{"type":"string","value":":method"},{"type":"string","value":"GET"}],[{"type":"string","value":":path"},{"type":"string","value":"/health?plugins"}],[{"type":"string","value":":scheme"},{"type":"string","value":"http"}],[{"type":"string","value":"accept"},{"type":"string","value":"/"}],[{"type":"string","value":"user-agent"},{"type":"string","value":"kube-probe/1.21"}],[{"type":"string","value":"x-forwarded-proto"},{"type":"string","value":"http"}],[{"type":"string","value":"x-request-id"},{"type":"string","value":"4b044a9a-730c-45df-8a9c-1c66df4cbfb0"}]]}],[{"type":"string","value":"host"},{"type":"string","value":"10.244.1.49:8282"}],[{"type":"string","value":"id"},{"type":"string","value":"5300754488508531884"}],[{"type":"string","value":"method"},{"type":"string","value":"GET"}],[{"type":"string","value":"path"},{"type":"string","value":"/health?plugins"}],[{"type":"string","value":"protocol"},{"type":"string","value":"HTTP/1.1"}],[{"type":"string","value":"scheme"},{"type":"string","value":"http"}]]}],[{"type":"string","value":"time"},{"type":"string","value":"2023-04-05T16:13:26.120151Z"}]]}],[{"type":"string","value":"source"},{"type":"object","value":[[{"type":"string","value":"address"},{"type":"object","value":[[{"type":"string","value":"socketAddress"},{"type":"object","value":[[{"type":"string","value":"address"},{"type":"string","value":"10.244.1.1"}],[{"type":"string","value":"portValue"},{"type":"number","value":59388}]]}]]}]]}]]}],[{"type":"string","value":"parsed_body"},{"type":"null","value":{}}],[{"type":"string","value":"parsed_path"},{"type":"array","value":[{"type":"string","value":"health"}]}],[{"type":"string","value":"parsed_query"},{"type":"object","value":[[{"type":"string","value":"plugins"},{"type":"array","value":[{"type":"string","value":""}]}]]}],[{"type":"string","value":"truncated_body"},{"type":"boolean","value":false}],[{"type":"string","value":"version"},{"type":"object","value":[[{"type":"string","value":"encoding"},{"type":"string","value":"protojson"}],[{"type":"string","value":"ext_authz"},{"type":"string","value":"v3"}]]}]],"level":"debug","msg":"Executing policy query.","query":"data.pebble.oceans.auth.allow","time":"2023-04-05T16:13:26Z","txn":5}
{"bundles":{"pebble/oceans":{}},"decision_id":"7d453acb-0174-470c-8737-8defad28cd2e","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.1.49","portValue":8282}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"10.244.1.49:8282",":method":"GET",":path":"/health?plugins",":scheme":"http","accept":"/","user-agent":"kube-probe/1.21","x-forwarded-proto":"http","x-request-id":"4b044a9a-730c-45df-8a9c-1c66df4cbfb0"},"host":"10.244.1.49:8282","id":"5300754488508531884","method":"GET","path":"/health?plugins","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-04-05T16:13:26.120151Z"},"source":{"address":{"socketAddress":{"address":"10.244.1.1","portValue":59388}}}},"parsed_body":null,"parsed_path":["health"],"parsed_query":{"plugins":[""]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"6c2529dd-2f13-4774-a933-7dd1e650fed2","version":"0.51.0-envoy"},"level":"info","metrics":{"timer_rego_external_resolve_ns":1216,"timer_rego_query_compile_ns":150626,"timer_rego_query_eval_ns":18727,"timer_server_handler_ns":1227823},"msg":"Decision Log","path":"pebble/oceans/auth/allow","result":null,"time":"2023-04-05T16:13:26Z","timestamp":"2023-04-05T16:13:26.123521589Z","type":"openpolicyagent.org/decision_logs"}
{"decision-id":"0bf96484-b421-4d62-946d-6f9461702ca8","level":"debug","msg":"no content-type header supplied, performing no body parsing","time":"2023-04-05T16:13:26Z"}

##################

opa-istio-config
apiVersion: v1
data:
config.yaml: |
services:
- name: controller
url: http://nginx-service
bundles:
pebble/oceans:
service: controller
resource: bundle.tar.gz
plugins:
envoy_ext_authz_grpc:
addr: :9191
path: pebble/oceans/auth/allow
decision_logs:
console: true

##########################################
Bundle Structure
. example.rego
. data.json

################################

example.rego

################################

package example

default allow = false

allow {
input.method = "GET"
input.path = [resource]
data.resources[resource]
}

################################

data.json

################################
{
"resources": {
"resource1": true,
"resource2": true,
"resource3": true
}
}

Answered by anderseknert

Apr 6, 2023

Right, that should not make a difference. Since you're getting false back, there must be something in the policy contributing to that decision. What it could be is hard to say without knowing what your policy looks like. I'd probably try opa eval with coverage, to try and see where things go wrong. If it's a single file, you can even use the Rego Playground with coverage turned on.

View full answer

anderseknert · 2023-04-06T09:37:12Z

anderseknert
Apr 6, 2023
Maintainer

Did you figure out what the issue was? @daniel-pebble

0 replies

daniel-pebble · 2023-04-06T11:29:48Z

daniel-pebble
Apr 6, 2023
Author

Sort of :D
I figured out the issue was with the path I was using for the bundle download, I've switched it to pebble/oceans/auth, and it now is loading the policy and evaluating it; however, the decision always resolves to a deny.
I've compared the "/v1/data" endpoint when running in bundle mode and push, and the data is the same, so I'm a bit puzzled as to why it's not working.

Looking at the "/v1/policies" endpoint, the only difference I can see is the id, but I can't see why that would cause an issue

config.yaml: |
    plugins:
      envoy_ext_authz_grpc:
        addr: :9191
        path: pebble/oceans/auth/allow
        dry-run: false
        enable-reflection: false
    services:
    - name: controller
      url: http://nginx-service/
    bundles:
      pebble/oceans/auth:
        service: controller
        resource: bundle.tar.gz
    decision_logs:
      console: true

{
  "bundles": {
    "pebble/oceans/auth": {}
  },
  "decision_id": "410fbf44-345c-4238-a4cc-e25051ee1215",
  "input": {
    "attributes": {
      "destination": {
        "address": {
          "socketAddress": {
            "address": "10.244.1.104",
            "portValue": 80
          }
        },
        "principal": "spiffe://cluster.local/ns/default/sa/deployment-service-account"
      },
      "metadataContext": {},
      "request": {
        "http": {
          "headers": {
            ":authority": "router.default.svc.cluster.local",
            ":method": "GET",
            ":path": "/api/deployment/2/kubeapps/audit/0?pageSize=1000",
            ":scheme": "http",
            "pbs-auth-jwt": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VySWQiOiIxODJlMTdhNC0yNTdkLTQ4NWItYTUxMS0wODY0NWM5ODdhZTEiLCJ1c2VyRGlyZWN0b3J5SWQiOiIxZDZkM2JmZi1lZDRjLTRiYTEtOThlMi1hMmYwYjM4YzFjMTUiLCJleHAiOjE2ODA5MDI1MjIsImlhdCI6MTY4MDc3MjkyMiwibmJmIjoxNjgwNzcyOTIyfQ.snCprQoF6me_YRlwXr-oWZCivLx5HcB0oNtt5s9x3Js",
            "traceparent": "00-614acff935880b647a5eb35263910768-fecb56a338d4d140-00",
            "x-b3-sampled": "0",
            "x-b3-spanid": "b0ec61697a8442a3",
            "x-b3-traceid": "a340e3cbc82e9350b0ec61697a8442a3",
            "x-envoy-attempt-count": "1",
            "x-forwarded-client-cert": "By=spiffe://cluster.local/ns/default/sa/deployment-service-account;Hash=6225607efb822202d62c384c8e4f8f3521bafa9be4b11ef49d0269e812c48b0a;Subject=\"\";URI=spiffe://cluster.local/ns/default/sa/deployment-service-account",
            "x-forwarded-proto": "http",
            "x-request-id": "6a280aa1-bd1c-4728-ab9f-ced509f8264c"
          },
          "host": "router.default.svc.cluster.local",
          "id": "211598686626230300",
          "method": "GET",
          "path": "/api/deployment/2/kubeapps/audit/0?pageSize=1000",
          "protocol": "HTTP/1.1",
          "scheme": "http"
        },
        "time": "2023-04-06T09:23:15.734448Z"
      },
      "source": {
        "address": {
          "socketAddress": {
            "address": "10.244.1.83",
            "portValue": 53326
          }
        },
        "principal": "spiffe://cluster.local/ns/default/sa/deployment-service-account"
      }
    },
    "parsed_body": null,
    "parsed_path": [
      "api",
      "deployment",
      "2",
      "kubeapps",
      "audit",
      "0"
    ],
    "parsed_query": {
      "pageSize": [
        "1000"
      ]
    },
    "truncated_body": false,
    "version": {
      "encoding": "protojson",
      "ext_authz": "v3"
    }
  },
  "labels": {
    "id": "303781b6-6620-48f0-9fb1-6844a7e4152f",
    "version": "0.51.0-envoy"
  },
  "level": "info",
  "metrics": {
    "timer_rego_external_resolve_ns": 1140,
    "timer_rego_query_eval_ns": 5481726,
    "timer_server_handler_ns": 5844963
  },
  "msg": "Decision Log",
  "path": "pebble/oceans/auth/allow",
  "result": false,
  "time": "2023-04-06T09:23:15Z",
  "timestamp": "2023-04-06T09:23:15.741102766Z",
  "type": "openpolicyagent.org/decision_logs"
}

3 replies

anderseknert Apr 6, 2023
Maintainer

Right, that should not make a difference. Since you're getting false back, there must be something in the policy contributing to that decision. What it could be is hard to say without knowing what your policy looks like. I'd probably try opa eval with coverage, to try and see where things go wrong. If it's a single file, you can even use the Rego Playground with coverage turned on.

Answer selected by daniel-pebble

daniel-pebble Apr 6, 2023
Author

I managed to find the issue using "eval". Turns out the bundle was created with a much older rego policy by mistake, after double-checking the policies endpoint I spotted the issue

anderseknert Apr 6, 2023
Maintainer

Great! Thanks for reporting back 😃

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Open Policy Agent

503 Service Unavailable - Using Bundle API #392

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Open Policy Agent

503 Service Unavailable - Using Bundle API #392

Uh oh!

Uh oh!

daniel-pebble Apr 5, 2023

example.rego

data.json

Replies: 2 comments · 3 replies

Uh oh!

anderseknert Apr 6, 2023 Maintainer

Uh oh!

Uh oh!

daniel-pebble Apr 6, 2023 Author

Uh oh!

anderseknert Apr 6, 2023 Maintainer

Uh oh!

daniel-pebble Apr 6, 2023 Author

Uh oh!

anderseknert Apr 6, 2023 Maintainer

daniel-pebble
Apr 5, 2023

Replies: 2 comments 3 replies

anderseknert
Apr 6, 2023
Maintainer

daniel-pebble
Apr 6, 2023
Author

anderseknert Apr 6, 2023
Maintainer

daniel-pebble Apr 6, 2023
Author

anderseknert Apr 6, 2023
Maintainer