chore: move from prom-client to OpenTelemetry metrics

[JessicaJHee]

Description

Adopts OpenTelemetry metrics for metrics and move away from using prom-client.

This change adds these new Backstage specific metrics:

• catalog.stitching.duration
• catalog.stitching.queue.length
• catalog.stitching.queue.length
• backend_tasks.task.runs.count
• backend_tasks.task.runs.duration

However, OpenTelemetry doesn’t collect Node.js specific metrics like nodejs_eventloop_lag_seconds.

• Note: Node.js specific metrics are WIP in this PR
• Edit: this has been merged and released, these missing metrics are now included with OpenTelemetry

The application metrics will be exposed at localhost:9464/metrics

• See sample log file: application.log

The host/process metrics will be exposed at localhost:9463/metrics

• See sample log file: host.log

Which issue(s) does this PR fix

• Fixes RHIDP-4404

Please make sure that the following steps are complete:

• GitHub Actions are completed and successful
• Unit Tests are updated and passing
• E2E Tests are updated and passing
• Documentation is updated if necessary (requirement for new features)
• Add a screenshot if the change is UX/UI related

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[github-actions]

The image is available at: quay.io/janus-idp/backstage-showcase:pr-1811!

[github-actions]

The image is available at:

• quay.io/janus-idp/backstage-showcase:pr-1811 or
• quay.io/janus-idp/backstage-showcase:pr-1811-720a38b0

[JessicaJHee]

TODO: need to update monitoring-and-logging.md, will go through this set up & update the docs

[github-actions]

The image is available at:

• quay.io/janus-idp/backstage-showcase:pr-1811 or
• quay.io/janus-idp/backstage-showcase:pr-1811-ad3afca1

[github-actions]

The image is available at:

• quay.io/janus-idp/backstage-showcase:pr-1811 or
• quay.io/janus-idp/backstage-showcase:pr-1811-dfeb8589

[github-actions]

The image is available at:

• quay.io/janus-idp/backstage-showcase:pr-1811 or
• quay.io/janus-idp/backstage-showcase:pr-1811-bf6ed0c8

[sonarqubecloud]

Quality Gate failed

Failed conditions
1 Security Hotspot

See analysis details on SonarQube Cloud

[github-actions]

The image is available at:

• quay.io/janus-idp/backstage-showcase:pr-1811 or
• quay.io/janus-idp/backstage-showcase:pr-1811-217a8d33

[kim-tsao]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kim-tsao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kim-tsao]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment