[jmx-scraper] Create correctness tests against target systems

[atoulme]

Using the golden tool, generate expected metric files containing metric output from typical target systems.

For correctness, the test consists of a docker compose setup with one container running the target system, such as cassandra, kafka, tomcat - and a container running golden.

The jmx-scraper then runs and outputs to golden the otlp metrics it captures. The golden image then checks the metrics received to the expected yaml.

See https://github.com/signalfx/opentelemetry-ibm-mq-monitoring/tree/main/golden for an example.

[SylvainJuge]

We already have integration tests that start a target system using testcontainers, could you elaborate how that would be different or better with this tool ? Can you link (or provide details) on a bug report where there was something not covered by existing tests ?

Because those integration tests do test the yaml definitions themselves, the testing is split/duplicated in both contrib and instrumentation with very similar testing infrastructure.
This means that the tests in the contrib repository will only contain the "legacy" yaml definitions that are close to the jmx-gatherer metrics, and that the aligned yaml definitions in instrumentation are tested there (this only covers a subset of target systems)

What is currently missing though is:

• proper testing of the captured values as we only assert for the metrics definitions (name, attributes and other properties)
• reusing/extending the metrics assertions provided by the SDK to use the same assertions as in other places

[atoulme]

That's what I have in mind: testing all the attributes and the completeness of the metrics, meaning, any new metrics or missing metrics. I also want to use a tool like golden or weaver so we can reuse those assets.