[Request] Document Developer Setup Docker local

[ollyhowell]

Draft tutorial from @danielkhan:

Developer Setup Docker local

[Intro … E.g. Elastic observability is free and open, can be installed locally, supports many languages and OpenTelemetry. […]
This documentation shows how to get started with Elastic Observability in 5 minutes. […]]
​
Install Get Docker | Docker Documentation and Install Docker Compose | Docker Documentation .
Please make sure to dedicate
​
[Alternatively to the instructions provided here, could we also provide some artefact on e.g. GitHub to simply clone?]
​
Create a directory elastic-observability somewhere on your system.
​
In this directory, create a file docker/compose.yml with this content:
​

version: '2.2'
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    environment:
    - bootstrap.memory_lock=true
    - cluster.name=docker-cluster
    - cluster.routing.allocation.disk.threshold_enabled=false
    - discovery.type=single-node
    - xpack.security.enabled=true
    - xpack.security.authc.api_key.enabled=true
    - ES_JAVA_OPTS=-Xms750m -Xmx750m
    ulimits:
      memlock:
        hard: -1
        soft: -1
    volumes:
    - esdata:/usr/share/elasticsearch/data
    ports:
    - 9200:9200
    networks:
    - esnet
    healthcheck:
      interval: 20s
      retries: 10
      test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"'
​
  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:${VERSION}
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_KIBANA_PASSWORD}
    ports:
    - 5601:5601
    networks:
    - esnet
    healthcheck:
      interval: 10s
      retries: 20
      test: curl -u ‘kibana_system:${ELASTICSEARCH_KIBANA_PASSWORD}’ —write-out ‘HTTP %{http_code}’ —fail —silent —output /dev/null http://localhost:5601/api/status
  
  apm-server:
    image: docker.elastic.co/apm/apm-server:${VERSION}
    depends_on:
      elasticsearch:
        condition: service_healthy
      kibana:
        condition: service_healthy
    cap_add: [“CHOWN”, “DAC_OVERRIDE”, “SETGID”, “SETUID”]
    cap_drop: [“ALL”]
    ports:
    - 8200:8200
    networks:
    - esnet
    command: >
       apm-server -e
         -E apm-server.rum.enabled=true
         -E setup.kibana.host=kibana:5601
         -E apm-server.kibana.username=elastic
         -E apm-server.kibana.password=${ELASTICSEARCH_PASSWORD}
         -E setup.template.settings.index.number_of_replicas=0
         -E apm-server.kibana.enabled=true
         -E apm-server.kibana.host=kibana:5601
         -E output.elasticsearch.hosts=[“elasticsearch:9200”]
         -E output.elasticsearch.username=“elastic”
         -E output.elasticsearch.password=${ELASTICSEARCH_PASSWORD}
​
    healthcheck:
      interval: 10s
      retries: 12
      test: curl —write-out ‘HTTP %{http_code}’ —fail —silent —output /dev/null http://localhost:8200/
​
volumes:
  esdata:
    name: esdata
    driver: local
​
networks:
  esnet:
    name: esnet
    driver: bridge

​
Additionally, create a file .env that will hold your configuration:
​

VERSION=7.16.0-SNAPSHOT // Needs to change depending on the doc page viewed
ELASTICSEARCH_KIBANA_PASSWORD=
ELASTICSEARCH_PASSWORD=

​
ATTENTION: When using Git, please make sure to add .env to your .gitignore file to not accidentally push credentials to GitHub.
​
Now in this directory, run docker compose up elasticsearch to download and start Elasticsearch.
Once Elasticsearch is running , run docker exec elasticsearch /bin/bash -c ‘bin/elasticsearch-setup-passwords auto —batch —url http://elasticsearch:9200'
​
This will create and store a set of passwords in Elastic.
Store the output of this command at a safe place.
​
Open your .env file and fill copy the password for thekibana_system and elastic user to their respective config variables:
​

VERSION=7.16.0-SNAPSHOT
ELASTICSEARCH_KIBANA_PASSWORD=<Generated password for user kibana_system>
ELASTICSEARCH_PASSWORD=<Generated password for user elastic>

​
Now stop docker and run it again using docker compose up (without elasticsearch).
This will start Elasticsearch, Kibana and the APM Server.
Please wait until bloglines for app-server show up on the console output.
Please ignore lines, that indicate an authentication error.
​

[I’m getting this - currently trying to figure out what it is - does not have an impact]]
elasticsearch | {"type": "server", "timestamp": "2021-11-25T12:26:05,969Z", "level": "INFO", "component": "o.e.x.s.a.RealmsAuthenticator", "cluster.name": "docker-cluster", "node.name": "8d05684f22de", "message": "Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]", "cluster.uuid": "BOGGeCrEQz-1vLjiII1NBQ", "node.id": "isSa2LI4Sa2mwmXxa63y1A"
​
Now navigate to http://localhost:5601 in your browser and log in with the username elastic and the password you generated before.
Click on the Observability tile and follow the instructions on screen to set up APM.
​
​
[In my tests, I saw that the Docker default memory setting wasn’t enough on my Silicon MBP and the Elastic server crashed with Error 137.
Can we verify and propose safe memory limits? ]

[bmorelli25]

I don't think we want to promote the APM Server binary as the best way for users to set up a Docker dev environment. I think the Elastic Stack + Elastic Agent would be better to document. As it is, this guide really only sets up APM, not Elastic Observability. Documenting Elastic Agent instead would fix that.

FWIW, we do already have documentation that is similar to this, although perhaps not as detailed (and certainly hasn't been updated in a while). The APM Server Legacy Quick Start: https://www.elastic.co/guide/en/apm/guide/current/quick-start-overview.html

[ollyhowell]

@danielkhan what do you think?

[danielkhan]

@bmorelli25 can you describe the setup you'd recommend to a local development setup?

Also, can we test the quick start you referenced and see what needs to be updated?
I created my guide months ago and as far as I remember, the solutions in our docs didn't work with 7.16. If I remember correctly, it was because security is enabled and required now by default.
If this guide does not work with 8.0, how can we ensure that such artifacts are kept up-to-date in the future, @ollyhowell?

[ollyhowell]

It sounds like there's an overlap between the above quick start and the APM one but neither are ideal / up-to-date.
I've not ran thru either setup myself yet but let's work out what the 'ideal' setup would be before you get too deep in this issue, @colleenmcginnis

To the general point about keeping docs up-to-date, the main problem is we're under resourced so there's a never ending To Do list and plenty of areas that don't get enough attention. We have talked about testing documentation and having 'upgrade checklists' for each release to run thru and spot these kind of issues. We don't have a great answer for this just now but it is on our radar and continuing to encourage devs, PMs etc. to flag and contribute to these issues is a no brainer.

[danielkhan]

How should we proceed with this?
@bmorelli25 brought up that we want to recommend a different setup. Who do we think is responsible for providing guidance in cases like this?
@alex-fedotyev, @simitt do you have a preferred way we should document here?
The goal is to have a quick and easy way to spin up a local elastic setup for o11y.

[simitt]

Ensuring that docs in the APM Server guide are up-to-date has been the responsibility of the APM Server team. I suggest we keep owning that for now. So ideally, whenever we update or implement a new feature, there is a dedicated docs issue that we share with the tech writer team and that gets scheduled, or if it is just tiny updates, engineers might include the changes directly in their PR.

It becomes a bit more tricky with Elastic Agent. I'd say that ideally the Elastic Agent team (@jlind23 ) has a comprehensive documentation with examples how to get started, and keeps these docs up to date.
APM Server should still have its own docs and examples, but link to the Elastic Agent docs as much as possible to avoid duplication and getting out of sync.

[simitt]

For the concrete ask to document docker with Elastic Agent and Fleet, we will need to provide extra guidance on how to add APM Server Integration to the Agent policy to which the Elastic Agent is enrolled.

[danielkhan]

I think this discussion shows an inherent problem.

This was the original claim by @bmorelli25

I don't think we want to promote the APM Server binary as the best way for users to set up a Docker dev environment. I think the Elastic Stack + Elastic Agent would be better to document.

For a dev setup, one would need to run Agent and Fleet Server in a Docker container each and, even with an ideal Docker file, then also get the configuration within Kibana right so that Agent and Fleet are enrolled and connected.
In this case, all components would run isolated in a Docker container. Does it provide any benefit compared to sending telemetry to the APM server?
I tried setting this up and it's really not easy.

As it is, this guide really only sets up APM, not Elastic Observability. Documenting Elastic Agent instead would fix that.

I really want this to be clarified. What would using Agent change in this particular case?

I can not see any benefit from using Agent for this particular use case and if we want to default to it we have to make it simple to set it up.

[bmorelli25]

As it is, this guide really only sets up APM, not Elastic Observability. Documenting Elastic Agent instead would fix that.

I really want this to be clarified. What would using Agent change in this particular case?

A few thoughts here. First — and Silvia, correct me if this has changed — APM Server standalone will soon be deprecated in favor of Elastic Agent + the APM integration. That's a good enough reason in itself to avoid documenting the standalone (now legacy) method of running APM. In addition, the Docker guide in this issue only sets up Elasticsearch, Kibana, and APM Server. It's a nice way to get started with Elastic APM, but it excludes the rest of Elastic Observability.

Using Elastic Agent, instead, would be more observability focussed — users can add the APM integration for APM, but also have access to our other Observability integrations: synthetics, host logs/metrics, etc.

[danielkhan]

A few thoughts here. First — and Silvia, correct me if this has changed — APM Server standalone will soon be deprecated in favor of Elastic Agent + the APM integration.

That's valid. Do we know who drives this decision and makes the call @simitt?

Using Elastic Agent, instead, would be more observability focussed — users can add the APM integration for APM, but also have access to our other Observability integrations: synthetics, host logs/metrics, etc.

The thing is that from a Docker container all you will see are the metrics from the container Agent runs on and also Synthetics isn't relevant in this scenario either.

[bmorelli25]

The thing is that from a Docker container all you will see are the metrics from the container Agent runs on and also Synthetics isn't relevant in this scenario either.

Maybe I'm misunderstanding the goal of this docker setup tutorial. Isn't this a way for users to easily test out Elastic Observability? To do that, don't we want to give them a playground where they can install whatever integrations they choose?

[danielkhan]

Cloud provides a viable playground for testing the whole range of observability.
This is for folks that want to use Elastic to mostly collect traces during development.
The purpose is to provide something similar as shown here: https://www.jaegertracing.io/docs/1.29/getting-started/
Run it, send traces and see them in Elastic. This can be either through an APM agent or OpenTelemetry. We support both very well and could be a viable alternative to Jaeger if we make it easy to set it up.

[bmorelli25]

Oh. In that case, let's start over.

A legacy (APM Server standalone) docker guide exists: https://www.elastic.co/guide/en/apm/guide/current/quick-start-overview.html. It's similar to what you have in this issue. We can update it and add context to turn it into a more useful developer guide. But there's a catch. Starting in 7.17, our documentation began focussing on the new method of running APM: Elastic Agent + the APM integration. Because of this integration focus, it wouldn't make sense for an APM Server standalone docker dev tutorial to live side-by-side with the other top-level docs in the APM Guide. A few reasons come to mind:

• Settings and configurations are different
• Feature support is different (ex: tail-based sampling is documented but not supported in legacy mode)
• Index lifecycle management is different
• Ingest pipelines are different

Like Silvia mentioned, I think we need a docker dev guide that is maintained by the Elastic Agent team but has instructions for how to enable and configure the APM integration. It might be more complex in the beginning, but it aligns better with where we're heading. (this is all assuming we still plan on deprecating APM Server in the near future)

[danielkhan]

I think this all makes sense. Ideally, this could also be an exercise to make the overall Agent/Fleet setup easier.
I know that there is some ongoing work. We might have to connect the strings here though. FYI @mukeshelastic