[Internal]: stack monitoring configuration guide inconsistencies / bugs

[shainaraskas]

Takeaways (See below)

added by @shainaraskas

• on this page: https://www.elastic.co/docs/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod
  • page send you off for sending data to a cluster, but the link is weirdly targeted to the stack monitoring overview rather than the stack options (here: https://www.elastic.co/docs/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed#collection-methods)
• on this page: https://www.elastic.co/docs/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat
  • possibly an invalid guidance for the scope property - when tested using the following spec, scope: node did not work for the reporter:
    • metricbeat installed on all clusters
    • scope: node

Actions

added by @shainaraskas

• Fix the link for the send data to your cluster step to point to the collection options
• Investigate metricbeat scope: node issue as described:
  • if the guidance is unclear/missing something, edit it
  • if it's a bug, log it with eng, determine if the steps need to be updated temporarily

Original description

From @thekofimensah in slack

I'm working on an implementation where a single elasticsearch cluster wants to be monitored by another elasticsearch instance using metricbeat.
From my troubleshooting, I used this reference and this reference.
Firstly for some feedback, the first doc somewhat suddenly ends by just saying to "send data to the cluster" without saying what you need to do to make it work. The second doc actually talks about this and it' not straight foreward. I would suggest referencing the second doc or writing it out in the first page for clarity.
Now for the second doc, there is something that I was having trouble with. It seems to me that if you use the default scope:node property and install metricbeat on all of the nodes, remote stack monitoring doesn't work. At least I wasn't able to make it work. However scope:cluster did make it work. If it's true that scope:node doesn't work for remote monitoring, I think it should explicitly say that.
The doc says:

Ideally, install a single Metricbeat instance configured with scope: cluster and configure hosts to point to an endpoint, such as a load-balancing proxy, which directs requests to the master-ineligible nodes in the cluster.
If this is not possible, then install one Metricbeat instance for each Elasticsearch node in the production cluster and use the default scope: node. When Metricbeat is monitoring Elasticsearch with scope: node then you must install a Metricbeat instance for each Elasticsearch node. If you don’t, some metrics will not be collected.
Metricbeat with scope: node collects most of the metrics from the elected master of the cluster, so you must scale up all your master-eligible nodes to account for this extra load. You should not use this mode if you have dedicated master nodes.

Point of contact.

Stakeholders: #stack-monitoring in slack