[Entitlements] Allow policy overrides via system properties

[ldematte]

This PR adds the ability to override entitlement policies for Elasticsearch plugins and modules via a system property.

The system property is in the form es.entitlements.policy.<plugin name>, and accepts a versioned policy:

versions:
  - version1
  - versionN
policy:
  <a standard entitlement policy>

For example:

versions:
  - 9.1.0
policy:
  ALL-UNNAMED:
    - set_https_connection_properties
    - outbound_network
    - files:
      - relative_path: ".config/gcloud"
        relative_to: home
        mode: read

The versioned policy needs to be base64 encoded. For example, to pass the above policy to a test cluster via gradle run:

./gradlew run --debug-jvm -Dtests.jvm.argline="-Des.entitlements.policy.repository-gcs=dmVyc2lvbnM6CiAgLSA5LjEuMApwb2xpY3k6CiAgQUxMLVVOTkFNRUQ6CiAgICAtIHNldF9odHRwc19jb25uZWN0aW9uX3Byb3BlcnRpZXMKICAgIC0gb3V0Ym91bmRfbmV0d29yawogICAgLSBmaWxlczoKICAgICAgLSByZWxhdGl2ZV9wYXRoOiAiLmNvbmZpZy9nY2xvdWQiCiAgICAgICAgcmVsYXRpdmVfdG86IGhvbWUKICAgICAgICBtb2RlOiByZWFkCg=="

The versions listed in the policy are string-matched against Build.version().current(); it is possible to specify any number of versions. If the list is empty/there is no versions field, the policy is assumed to match any Elasticsearch versions.

The override policy specified for any given plugin replaces the embedded policy for that plugin. See how EntitlementsAllowedViaOverrideIT replaces an empty policy for the entitlement-test-plugin with a policy that allows load_native_libraries and access to files in the test read_dir.

Also tested manually with an override with a different version, with an override with an invalid policy and with a valid override (see command above).

Relates to ES-11009

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[prdoyle]

Remind me, what was the motivation for the versions?

I see from the Jira:

It can have a version so we can blow up if it’s too old (we solved it) or ignore it

If we, in essence, just want these to "expire" at some point, then a list of patch versions could be needlessly cumbersome. It also raises questions about how this would work for serverless.

[ldematte]

If we, in essence, just want these to "expire" at some point, then a list of patch versions could be needlessly cumbersome. It also raises questions about how this would work for serverless.

This is why I made version non-semantic, just a (list) of versions.
I would imagine that the workflow would be: we find a problem with 9.0.0, mitigate that with an override, fix it in 9.0.1
This works with serverless 2, in 2 ways:

• We list the build hash of the deployed versions we know we need to patch (that's why I made this a list)
• Or we omit versions and the policy applies to all (which would also work with serverless)

[mark-vieira]

If we, in essence, just want these to "expire" at some point, then a list of patch versions could be needlessly cumbersome. It also raises questions about how this would work for serverless.

I don't think we'd use this mechanism there. We'd just do an actual patch release.

[mark-vieira]

• We list the build hash of the deployed versions we know we need to patch (that's why I made this a list)

I don't think it's worth supporting this case. As mentioned before, we can just patch, or use the "versionless" format. We don't have to worry about this stale configuration being mistakenly kept around even when a fix is deployed because we own the deployment here.

[mark-vieira]

I would imagine that the workflow would be: we find a problem with 9.0.0, mitigate that with an override, fix it in 9.0.1
This works with serverless 2, in 2 ways:

👍 Presumably whatever we "patch" here will be fixed in the next release. So there's no need to be lenient and allow future patch versions.

[ldematte]

Presumably whatever we "patch" here will be fixed in the next release. So there's no need to be lenient and allow future patch versions.

++
When I thought about multiple versions, I though they might be useful for specifying multiple past versions... but since these are applied ad-hoc in the end, probably a single version (plus "versionless") would be enough

[ldematte]

Presumably whatever we "patch" here will be fixed in the next release. So there's no need to be lenient and allow future patch versions.

Should I remove the ability to specify multiple versions and have just one version?

[prdoyle]

IMHO it's hard to imagine regretting support for multiple versions.

[prdoyle]

I don't think we'd use this mechanism there. We'd just do an actual patch release.

In serverless, this would be faster than a patch release because there's no need a full CI test cycle, right? I think we want to be careful about limiting our options during an incident.

[mark-vieira]

When I thought about multiple versions, I though they might be useful for specifying multiple past versions... but since these are applied ad-hoc in the end, probably a single version (plus "versionless") would be enough

Got it. This works in the case where we release 9.0.1 or 8.18.0.1 and find an issue. I'm good with keeping this flexability. We can always just not leverage it.

[mark-vieira]

In serverless, this would be faster than a patch release because there's no need a full CI test cycle, right? I think we want to be careful about limiting our options during an incident.

Good point, it would be a considerably faster remediation. I'm good with allowing "versionless" patches just to give us that option.

[jdconrad]

In the interest of time I think we should get this change in.

I've left a design change suggestion as a comment in the parser. Please let me know your thoughts -- could be a follow up if you think it would be a positive change.

[jdconrad]

My suggestion would be to not have the version parsing as part of the PolicyParser at all since it's only used to determine if a policy override is valid.

I see two different possibilities to accomplish this:

1. Change PolicyParser to take in an XContentBuilder instead of an InputStream. Pull out the xcontent necessary for version checking in elasticsearch bootstrapping code.
2. Change the override to only be JSON for the policy and some stream of bytes for the versions. You can then read that stream of bytes up to where the policy begins and then pass the remainder of the InputStream into the PolicyParser.

I think this may also remove the need for a VersionedPolicy record as well.

[ldematte]

Thanks @jdconrad for the suggestions

Change PolicyParser to take in an XContentBuilder instead of an InputStream. Pull out the xcontent necessary for version checking in elasticsearch bootstrapping code.

I actually started with that but made things more complicated. I will give it another go though.

Change the override to only be JSON for the policy and some stream of bytes for the versions. You can then read that stream of bytes up to where the policy begins and then pass the remainder of the InputStream into the PolicyParser.

I don't understand how this would work: currently the parser has to be YAML, not JSON. We can change it, but then it would be strange/cumbersome to pass a JSON + something via the command line.
Maybe we could have something like a JSON holding the version + the encoded policy, like base64({ versions: [], policy: "<base64 policy>"}). I don't like the double encoding and change of format too much but...

Maybe using JSON for such a simple object can be passed inline

-Des.entitlements.policy.repository-gcs='{ versions: [], policy: "dmVyc2lvbnM6CiAgLSA5Lj..." }'

does not look so bad.
I think I like the first alternative more, the second still look a bit messy to me, but wdyt?
(and I think this is for a follow-up I think, I'll merge this as is)

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.18
✅	8.x
✅	9.0

[jdconrad]

@ldematte Follow up is totally fine with me. I actually mean YAML 🤦 so I understand how my comment would have been confusing :)