OPA v0.38.0 Release

[peteroneilljr]

This release contains a number of fixes and enhancements.

It contains one backwards-incompatible change to the JSON representation
of metrics in Status API payloads, please see the section below.

Rich Metadata

It is now possible to annotate Rego policies in a way that can be
processed programmatically, using Rich Metadata.

# METADATA
# title: My rule
# description: A rule that determines if x is allowed.
# authors:
# - Jane Austin <[email protected]>
allow {
  ...
}

The available keys are:

• title
• description
• authors
• organizations
• related_resources
• schemas
• scope
• custom

Custom annotations can be used to annotate rules, packages, and
documents with whatever you specifically need, beyond the generic
keywords.

Annotations can be retrieved using the Golang library
or via the CLI, opa inspect -a.

All the details can be found in the documentation on Annotations.

Every Keyword

A new keyword for explicit iteration is added to Rego: every.

It comes in two forms, iterating values, or keys and values, of a
collection, and asserting that the body evaluates successfully for
each binding of key and value to the collection's elements:

every k, v in {"foo": "FOO", "bar": "BAR" } {
  upper(k) == v
}

To use it, import future.keywords.every or future.keywords.

For further information, please refer to the Every Keyword docs
and the new section on FOR SOME and FOR ALL in the Intro docs.

Tooling, SDK, and Runtime

• Compile API: add disableInlining option (#4357) reported and fixed by @srlk
• Status API: add http_code to response (#4259) reported and fixed by @jkbschmid
• Status plugin: publish experimental bundle-related metrics via prometheus endpoint (authored by @rafaelreinert) -- See Status Metrics for details.
• SDK: don't panic without config (#4303) authored by @damienjburks
• Storage: Support index for array appends (for JSON Patch compatibility)
• opa deps: Fix pretty printed output to show virtual documents (#4342)

Rego and Topdown

• Parser: parse 'with' on 'some x in xs' expression (#4226)
• AST: hash containers on insert/update (#4345), fixing a data race reported by @skillcoder
• Planner: Fix bug related to undefined results in dynamic lookups

Documentation and Website

• Policy Reference: update EBNF to include "every" and "some x in ..." (#4216)
• REST API: Update docs on 400 response
• README: Include Google Analytic Instructions
• Envoy primer: use variables instead of objects
• Istio tutorial: expose application to outside traffic
• New "Community" Webpage (authored by @msorens)

WebAssembly

• OPA now uses Wasmtime 0.34.0 to evaluate its Wasm modules.

Miscellaneous

• Build: make build now builds without errors (by disabling Wasm) on darwin/arm64 (M1)
• Various dependency bumps.
  • OpenTelemetry SDK: 1.4.1
  • github.com/prometheus/client_golang: 1.12.1

Backwards incompatible changes

The JSON representation of the Status API's payloads -- both for GET /v1/status
responses and the metrics sent to a remote Status API endpoint -- have changed:

Previously, they had been serialized into JSON using the standard library "encoding/json"
methods. However, the metrics coming from the Prometheus integration are only available
in Golang structs generated from Protobuf definitions. For serializing these into JSON,
the standard library functions are unsuited:

• enums would be converted into numbers,
• field names would be snake_case, not camelCase,
• and NaNs would cause the encoder to panic.

Now, we're using the protobuf ecosystem's jsonpb package, to serialize the Prometheus
metrics into JSON in a way that is compliant with the Protobuf specification.

Concretely, what would before be

  "metrics": {
    "prometheus": {
      "go_gc_duration_seconds": {
        "help": "A summary of the GC invocation durations.",
        "metric": [
          {
            "summary": {
              "quantile": [
                {
                  "quantile": 0,
                  "value": 0.000011799
                },
                {
                  "quantile": 0.25,
                  "value": 0.000011905
                },
                {
                  "quantile": 0.5,
                  "value": 0.000040002
                },
                {
                  "quantile": 0.75,
                  "value": 0.000065238
                },
                {
                  "quantile": 1,
                  "value": 0.000104897
                }
              ],
              "sample_count": 7,
              "sample_sum": 0.000309117
            }
          }
        ],
        "name": "go_gc_duration_seconds",
        "type": 2
      },

is now:

  "metrics": {
    "prometheus": {
      "go_gc_duration_seconds": {
        "name": "go_gc_duration_seconds",
        "help": "A summary of the pause duration of garbage collection cycles.",
        "type": "SUMMARY",
        "metric": [
          {
            "summary": {
              "sampleCount": "1",
              "sampleSum": 4.1765e-05,
              "quantile": [
                {
                  "quantile": 0,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 0.25,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 0.5,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 0.75,
                  "value": 4.1765e-05
                },
                {
                  "quantile": 1,
                  "value": 4.1765e-05
                }
              ]
            }
          }
        ]
      },

Note that sample_count is now sampleCount, and the type is using the enum's
string representation, "SUMMARY", not 2.

Note: For compatibility reasons (the Prometheus golang client doesn't use the V2
protobuf API), this change uses jsonpb and not protojson.