test: Improve the performance of the OPA rego rules

[siegfriedweber]

Description

Improve the performance of the OPA rego rules

• The with keyword in the batch rules prohibited the memoization of results. It is now replaced with function calls where the altered input action is passed as parameter.
• The policies that match a user or group, are only computed once in a batch query.

The performance was tested with a FilterTables query with 10.000 filterResources, 19 catalog policies, 24 schema policies and 34 table policies.

Before the optimization:

"metrics": {
  "timer_rego_load_bundles_ns": 36961138,
  "timer_rego_module_compile_ns": 89543750,
  "timer_rego_module_parse_ns": 33566595,
  "timer_rego_query_compile_ns": 149432,
  "timer_rego_query_eval_ns": 220644049527,
  "timer_rego_query_parse_ns": 107302
}

Query time: 221 seconds

After the optimization:

"metrics": {
  "timer_rego_load_bundles_ns": 34996456,
  "timer_rego_module_compile_ns": 122469832,
  "timer_rego_module_parse_ns": 33778511,
  "timer_rego_query_compile_ns": 52629,
  "timer_rego_query_eval_ns": 4283147151,
  "timer_rego_query_parse_ns": 41207
}

Query time: 4 seconds

The actual performance improvement depends on the query, policies and available hardware resources, the OPA version and so on.

Definition of Done Checklist

Author

• Integration tests passed (for non trivial changes)

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[siegfriedweber]

--- PASS: kuttl (927.07s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/opa-authorization_trino-451_hive-latest-4.0.1_opa-1.4.2_keycloak-25.0.0_openshift-false (462.47s)
        --- PASS: kuttl/harness/opa-authorization_trino-476_hive-latest-4.0.1_opa-1.4.2_keycloak-25.0.0_openshift-false (489.91s)
        --- PASS: kuttl/harness/opa-authorization_trino-470_hive-latest-4.0.1_opa-1.4.2_keycloak-25.0.0_openshift-false (464.58s)

[maltesander]

Thanks, LGTM!

[sbernauer]

Thanks for working on this, amazing work!
Would you mind pasting the final OPA eval benchmarks before and after in the description?

[siegfriedweber]

Would you mind pasting the final OPA eval benchmarks before and after in the description?

done

[sbernauer]

Awesome, thanks!