[Stack Monitoring] Create pre-configured elastic-package environment

[klacabane]

Summary

We can have a shareable configuration (maybe profiles?) that can be launched with elastic-package to set all the stack monitoring components with their corresponding integrations already installed and running on the agent. We end up with a SM UI loaded with data and ready to debug

This would be required in addition to the components already setup by elastic-package:

• start a logstash service running at a least a pipeline
• kibana with pre-installed integrations - we can use xpack.fleet.packages setting
• the integrations are attached to the running Elastic-Agent policy - we should be able to hook into the definition of the policy, but didn't look where it was

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[klacabane]

@crespocarlos this would be an extension of your approach to set kibana logs in elastic/integrations#3622, do you see any blockers to get everything setup in a profile ?

[klacabane]

For the package installation we can create a custom profile with elastic-package profiles create stack_monitoring --from default and add the integrations in the ~/.elastic-package/profiles/stack_monitoring/stack/kibana.config.8x.yml, both under the xpack.fleet.packages and xpack.fleet.agentPolicies[1].
It doesn't look like there is an ootb way to distribute these profiles, I'm thinking we could zip it and store it in publicly accessible google drive, or maybe a repo. Then maybe add a command to elastic-package to import them ?

[1] example for the elasticsearch package

xpack.fleet.packages:
  - name: system
    version: latest
  - name: elastic_agent
    version: latest
  - name: fleet_server
    version: latest
  - name: elasticsearch
    version: latest
xpack.fleet.agentPolicies:
  - name: Elastic-Agent (elastic-package)
    id: elastic-agent-managed-ep
    is_default: true
    is_managed: false
    namespace: default
    monitoring_enabled:
      - logs
      - metrics
    package_policies:
      - name: elasticsearch-1
        id: default-elasticsearch
        package:
          name: elasticsearch
        inputs:
          - type: elasticsearch/metrics
            enabled: true
            vars:
              - name: hosts
                value:
                  - https://elasticsearch:9200
              - name: username
                value: elastic
              - name: password
                value: changeme

[crespocarlos]

No blockers. I think we have 2 options here, either have that in a profile or have Kibana as a service in _dev/deploy, connecting to elastic-package stack. Then whenever we need to test elastic/integrations#3622 we would have a separate a dedicated Kibana for it.

In terms of configuration effort, it'd best to have it in a profile. Logs enabled by default seems more convenient. Besides, makes more sense to me if elastic-package service contains only services that are not part of the stack.

[klacabane]

I've created an ER in the elastic-package repo to discuss remote profiles elastic/elastic-package#923

In the meantime we can have a cheap workaround where we store these profiles in our public google drive and curl | unzip them

[matschaffer]

I noticed there's also an elastic-package install command, could we maybe use this to set up required packages on the default profile?

One gotcha I hit with the zipped profiles when testing logstash is that the monitored service (from elastic-package service up) might have a different hostname depending on the docker-compose version. v2 sticks to hyphens, v1 has some underscores.

Using a script that runs install might be able to avoid this by asking docker-compose for the hostname of a given logical service.

[matschaffer]

have that in a profile or have Kibana as a service in _dev/deploy, connecting to elastic-package stack

Just my ¥2, but I think a kibana service in _dev/deploy is a better fit for how a typical package works. If we test our packages by using the in-profile stack directly, we create a "special case" when compared to something like the nginx or mysql package.

[klacabane]

Just checked the install command and it only sets up the assets, we still have to configure the integration and attach it to the agent's policy which is what the profile approach was trying to solve (automate recurring and error-prone operations), but we could possibly enhance that command to also create the integration.

I think a kibana service in _dev/deploy is a better fit for how a typical package works

Agreed we should create dedicated services in each packages like logstash. we can add these as dependencies of the system tests (elastic/integrations#4008)

Using a script that runs install might be able to avoid this by asking docker-compose for the hostname of a given logical service.

Sounds like more robust approach. In addition to the container hostnames the script would have access to the configuration output by elastic-package stack shellinit which contains the necessary information to do a package installation and attach it to a policy

[matschaffer]

Sweet! I'm thinking we should close this and open a new issue to work on install-based automation. Any opposed?

[klacabane]

Follow up here elastic/integrations#4061