feat: add possibility to define custom attributes for runtime metrics

[HeavyPunk]

Hello, this PR add support for user defined metric attributes to runtime metrics (instruments/runtime package).
For example:
On latest release runtime metrics look somehow like this:
go_memory_used_bytes{go_memory_type="other"} 1.4046472e+07
On this PR you can enrich metrics by some attributes like this:
go_memory_used_bytes{environment="local", host="localhost", go_memory_type="other"} 1.4046472e+07

[linux-foundation-easycla]

• ❌ - login: @HeavyPunk / name: HeavyPunk . The commit (286537d) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

[dmathieu]

Couldn't you do the same with resources, and possibly views?

[dashpole]

Resource attributes are the proper place for this information

[dashpole]

Views can only filter attributes. We do have a labeler concept in otelhttp that is similar, but I don't think that applies to runtime metrics

[HeavyPunk]

So, i tried to use resources but attributes from resources published only at metadata metrics. The prometheus exporter has a good visualization for this. Look at these metrics please:
prometheus-exporter:

...
go_config_gogc_percent{otel_scope_name="go.opentelemetry.io/contrib/instrumentation/runtime",otel_scope_schema_url="",otel_scope_version="0.63.0"} 100
# HELP target_info Target metadata
# TYPE target_info gauge
target_info{my_key="my_value"} 1
...

When i need to get the go_config_gogc_percent metric for example, i need to make a query with joining metrics by intersect attributes. For example above it may be this query on promql:

go_config_gogc_percent
  * on(instance) group_left(my_key)
    target_info{my_key="my_value"}

But by first hand this query looks more difficult than just go_config_gogc_percent{my_key="my_value"} and by other hand each metric should have at least one equal attribute with metadata metric but there are no such attributes here.

So, if i wrong, explain me please how it should works

[dashpole]

That is basically correct.

each metric should have at least one equal attribute with metadata metric but there are no such attributes here.

target_info joins should be done on the job and instance labels. See https://prometheus.io/docs/guides/opentelemetry/#including-resource-attributes-at-query-time

[HeavyPunk]

Okay, then all exported metrics (runtime too) should contain job/instance attributes to be matched with target_info, am i right?
How these attributes should be added to runtime metrics correctly?

[HeavyPunk]

Oh, i see...

For each of a resource's OTel metrics, Prometheus converts it to a corresponding Prometheus time series, and (if target_info is generated) adds the right instance and job labels.

[dashpole]

job and instance labels are added by Prometheus after scraping based on the scrape_config defined in your prometheus configuration

[HeavyPunk]

Thank you for help! How do you think may this PR be useful for other cases?