[Fleet] Enrich agent with agent policies

[nchaulet]

Summary

Proof of concept of enrich agent with agent policies data (policy name, package_policies, ...) This allow to search agents by their agent policies property.

Need that elasticsearch PR to create related mappings. ingest pipeline and give more permissions.

Implementation details

To be able to implement that feature I introduced:

• .fleet-agent-policies-metadata a new index that contains the agent policy data we went to add to agents, that index is populated in deployPolicies the same place that populate .fleet-policies on every agent or package policy update
• fleet-agents-enrich-agent-policies a new enrich policy that use .fleet-agent-policies-metadata to create the enrich index
• fleet-agents@enrich-agent-policies-pipeline that use the enrich policy to populate the agent_policy field on the agent.

When a user deploy a policy (after a package or an agent policy update) in addition to create the existing .fleet-policies document, we now:

• create a new document in .fleet-agent-policies-metadata
• Execute the enrich policy
• Update the existing agent with the new agent policy data

How to run locally?

Run es from source PR need openjdk@21

yarn es  source -E xpack.security.authc.api_key.enabled=true -E xpack.security.authc.token.enabled=true  --source-path=/<path_to_your_elasticsearch_repo>

Enable the feature flag in your kibana.dev.yml

xpack.fleet.enableExperimental:
  [
    'enrichAgentPolicies'
  ]

Questions? Next steps

• Performance impact?
• Do we want to enrich with more data? outputs, ...
• We will be able to simplify the runtime field for agent status as there is no need to inject all agent policies timeout anymore.

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!

[juliaElastic]

Tested and works as expected, enriching .fleet-agents with agent policy data.
One potential bottleneck is updating the agent docs on scale and encountering version conflicts when the same agents are being updated by fleet-server on checkin/ack.

I'm wondering if we could use another index to store the enriched agent docs (with agent.id and agent_policy fields) and use an API that queries from both indices when fetching/filtering data for the Agent list UI.

{
        "_index": ".fleet-agents-7",
        "_id": "_2jehpMB8iUsJ7N-ehmW",
        "_score": 1,
        "_source": {
          "policy_coordinator_idx": 1,
          "agent": {
            "id": "agent1",
            "version": "8.10.2"
          },
          "enrolled_at": "2023-10-01T14:27:11Z",
          "policy_id": "fleet-server-policy",
          "policy_revision": 2,
          "last_checkin_status": "online",
          "active": true,
          "type": "PERMANENT",
          "last_checkin": "2023-10-01T14:31:29Z",
          "upgrade_started_at": "2023-10-01T14:29:57Z",
          "packages": [],
          "updated_at": "2023-10-01T14:31:35Z",
          "policy_revision_idx": 2,
          "agent_policy": {
            "package_policies": [
              {
                "name": "fleet_server-1",
                "namespace": "default",
                "package": {
                  "name": "fleet_server",
                  "version": "1.5.0"
                },
                "id": "fleet-server-policy"
              },
              {
                "name": "system-1",
                "namespace": "default",
                "package": {
                  "name": "system",
                  "version": "1.62.1"
                },
                "id": "fleet-server-policy"
              }
            ],
            "name": "Fleet Server Policy",
            "namespace": "default",
            "id": "fleet-server-policy",
            "inactivity_timeout": 1209600,
            "is_managed": false
          },
          "id": "agent1",
          "upgraded_at": null
        }
      }

[nchaulet]

One potential bottleneck is updating the agent docs on scale and encountering version conflicts when the same agents are being updated by fleet-server on checkin/ack.

I think they may be a an optimization here that will be to explicitly not use the pipeline in fleet server, when performing checkin and ack agent updates, (probably worth testing if this as a big performance impact before)