Integrate Camunda metrics into Spring Boot Actuator registry

[ThorbenLindhauer]

This issue was imported from JIRA:

Field	Value
JIRA Link	CAM-14729
Reporter	@tmetzke
Has restricted visibility comments	true

---

User Story (Required on creation):

• As a System Administrator, I want to see exposed engine metrics as part of all the metrics (for Camunda Spring Boot Starter or Camunda Run) accessible through the Spring Boot Actuator so that I can monitor the system.
• As an Operations Engineer, I can see the metrics my System Administrator exposed for me, to indicate the system's health.

Functional Requirements (Required before implementation):

• I can use the Micrometer metrics facades described in Actuator to monitor Camunda performance and usage metrics

Technical Requirements (Required before implementation):

• Add monitoring metrics to CamundaBpmActuatorConfiguration
• Fetch metrics from configured engine collector via API in a configurable interval (can be a different interval than the internal metrics DbReporter’s interval)
• Add collected metrics to Actuator’s Micrometer registry

Limitations of Scope (Optional):

Hints (optional):

• We already have a Camunda Actuator auto-configuration, defining health endpoints for the engine and the job executor

Links:

• https://github.com/camunda/camunda-bpm-platform-maintenance/issues/1831
• depends on https://jira.camunda.com/browse/CAM-14727

camunda-bpm-platform PR

• feat(engine): Micrometer metrics #4519

### camunda-docs-manual PR

[punitdarira]

Hi @ThorbenLindhauer!
I've drafted a PR.
For now all the metrics are at the root level.
I feel we should only expose Camunda metrics in actuator when it's enabled via properties or other means. What do you think?

[psavidis]

Hello @punitdarira ,

Thank you for considering this contribution.
Can i ask you to provide me with the steps you followed to test your code?

Petros

[punitdarira]

@psavidis ,
First I built the project of camunda-bpm-spring-boot-starter using mvn clean install -Dmaven.test.skip=true
Then I ran camunda-bpm-spring-boot-starter-rest using mvn clean spring-boot:run

We still have few things to implement like

• Taking the time interval for pushing the metrics in the registry from the user.
• Maybe making this whole feature disabled by default and only enabling when the property is set?
• Documentation

[psavidis]

Hey @punitdarira,

Exactly, you can continue with the above contributions if you wish.

• I will also follow up on the code review about the configurable fetch metrics interval
• About the point for disabling the feature by default, i'm still skeptical if we need a configuration. It sounds reasonable. I will discuss this internally with the team and let you know.
• When the above point is finalised, we will have an idea where to add the relevant documentation

[psavidis]

@psavidis , First I built the project of camunda-bpm-spring-boot-starter using mvn clean install -Dmaven.test.skip=true Then I ran camunda-bpm-spring-boot-starter-rest using mvn clean spring-boot:run

We still have few things to implement like

• Taking the time interval for pushing the metrics in the registry from the user.
• Maybe making this whole feature disabled by default and only enabling when the property is set?
• Documentation

I've tried to use your suggestion on running the spring boot maven plugin in the camunda-bpm-spring-boot-starter-rest and i'm getting a failure:

[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:3.2.2:run (default-cli) on project camunda-bpm-spring-boot-starter-rest: Unable to find a suitable main class, please add a 'mainClass' property -> [Help 1]

[punitdarira]

@psavidis , First I built the project of camunda-bpm-spring-boot-starter using mvn clean install -Dmaven.test.skip=true Then I ran camunda-bpm-spring-boot-starter-rest using mvn clean spring-boot:run
We still have few things to implement like

• Taking the time interval for pushing the metrics in the registry from the user.
• Maybe making this whole feature disabled by default and only enabling when the property is set?
• Documentation

I've tried to use your suggestion on running the spring boot maven plugin in the camunda-bpm-spring-boot-starter-rest and i'm getting a failure:

[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:3.2.2:run (default-cli) on project camunda-bpm-spring-boot-starter-rest: Unable to find a suitable main class, please add a 'mainClass' property -> [Help 1]

Sorry I missed mentioning that I added a Main.java in camunda-bpm-spring-boot-starter-rest. This is a simple Main.java
package org.camunda.bpm.spring.boot.starter.rest;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class Main {
public static void main(String... args) {
SpringApplication.run(Main.class, args);
}
}

I also changed the scope of com.h2database.h2 from test to compile in pom.xml of camunda-bpm-spring-boot-starter-rest project.

[punitdarira]

Hi @psavidis ,
Added changes for pickup up interval seconds from properties file

[psavidis]

Hello @punitdarira,

The analysis of this ticket is still on going by the team. When the last details are finalised, i will update this thread with the technical proposal and the decision making thread to make it transparent to you as well.

Till then, i'd recommend to pause the code contribution so that you don't make changes that might be reverted later.

Kind regards,
Petros

[psavidis]

Technical Proposal

Property Control

A. Add new Config

• A.1 Add a new Config param that controls, enables / disables the availability of the engine metrics to Spring Actuator.

  • A.1.1 In the property absence, the feature is disabled.

    • + User Control over the Metrics; no unnecessary business metrics are exposed (security)
    • + The property's absence is intuitive from a user perspective (no metric exposure if the property is missing)
    • + Backwards Compatibility; existing behaviour will be unaffected
    • - An additional property is introduced

  • A.1.2 In the property absence, the feature is enabled.

    • + User Control over the Metrics; no unnecessary business metrics are exposed (security)
    • - The property's absence is not intuitive from a user perspective
    • - An additional property is introduced

B. Do not Add new Config

• B.1 Enable the availability of the Metrics by default when Spring Boot Actuator is added on the classpath
  • + No additional property will be introduced
  • - The new metrics availability on actuator will not be controlled (security)

Recommendation: Favouring Security and Backwards Compatibility over Maintenance

➡ Decision: A.1.1 is chosen.

New Properties - Activation and Interval Configuration

A. Can we combine the activation flag with an interval property? (for example, by default, the value is null or -1, meaning it’s off.)

• A.1. The activation flag and property are combined
  • + We win adding and maintaining one extra property
  • - The convention does controls more than one thing
• A.2. The activation flag and property are separate
  • + Each flag does one thing (Single responsibility)
  • - Maintenance Overhead **(**One more flag needs to be added)

Recommendation: Favoring Maintenance Convenience over Single Responsibility, A.1 is recommended.

• Reasoning: It is a popular pattern that many frameworks use. Also, the scenario with the enablement flag disabled and a configured interval can lead the user to confusion.

Existing Metric Properties

There are existing properties that enable / disable the db reporting. (camunda-engine-properties#Metrics)

Should it control the engine metrics availability to the spring-actuator as well ?\

Answer: camunda.bpm.metrics.enabled will not be related to spring actuator at all.

Due to legacy, any confusion that might be introduced by the existing properties cannot be avoided.

Property Name and Default Value

Recommended Property Name: camunda.bpm.metrics.actuator.interval

Default Value: -1 (int)

Validation: A SpringBootStarterException will be thrown if the user’s input is less than -1. A SpringBootStarterException will be thrown for inputs equal to 0.

Reasoning for Selection:

1. Simplicity
2. Convey relation to Spring-Boot
3. Follows the convention of the unit of spring boot (not mentioned e.g with a suffix like seconds) (see spring actuator properties)

Documentation

Recommendation: Add the new property under the Metrics Area. Mention it is a spring boot specific property, its function, default value, erroneous values.
Moreover, highlight performance implications this property can have when configured with a small value due to the frequent SQL query executions it can produce.

[psavidis]

Hello @punitdarira,

I've posted the technical proposal after the analysis of the team for transparency sake.
Please let me know if you have any questions.

If not, I recommend to continue the code contribution according to the technical proposal and use it as a reference for the code review.

Let me know your thoughts.

Best,
Petros