Put component-identifying scope attributes for internal metrics back behind feature gate (open-telemetry#12933)

#### Context

PR open-telemetry#12617 introduced logic to inject new instrumentation scope
attributes in all internal telemetry to identify which Collector
component it came from. These attributes had already been added to
internal logs as regular log attributes, and this PR switched them to
scope attributes for consistency. The new logic was placed behind an
Alpha stage feature gate, `telemetry.newPipelineTelemetry`.

Unfortunately, the default "off" state of the feature gate disabled the
injection of component-identifying attributes entirely, which was a
regression since they had been present in internal logs in previous
releases. See issue open-telemetry#12870 for an in-depth discussion of this issue.

To correct this, PR open-telemetry#12856 was filed, which stabilized the feature gate,
making it on by default, with no way to disable it, and removed the
logic that the feature gate used to toggle. This was thought to be the
simplest way to mitigate the regression in the "off" state, since we
planned to stabilize the feature eventually anyways.

Unfortunately, it was found that the "on" state of the feature gate
causes a different issue: [the Prometheus
exporter](https://github.com/open-telemetry/opentelemetry-go/tree/main/exporters/prometheus)
is the default way of exporting the Collector's internal metrics,
accessible at `collector:8888/metrics`. This exporter does not currently
have any support for instrumentation scope attributes, meaning that
metric streams differentiated by said attributes but not by any other
identifying property will appear as aliases to Prometheus, which causes
an error. This completely breaks the export of Collector metrics through
Prometheus under some simple configurations, which is a release blocker.

#### Description

To fix this issue, this PR sets the `telemetry.newPipelineTelemetry`
feature gate back to "Alpha" (off by default), and reintroduces logic to
disable the injection of the new instrumentation scope attributes when
the gate is off, but only in internal metrics. Note that the new logic
is still used unconditionally for logs and traces, to avoid
reintroducing the logs issue (open-telemetry#12870).

This should avoid breaking the Collector in its default configuration
while we try to get a fix in the Prometheus exporter.

#### Link to tracking issue
No tracking issue currently, will probably file one later.

#### Testing

I performed some simple manual testing with a config file like the
following:

```yaml
receivers:
  otlp: [...]
processors:
  batch:
exporters:
  debug: [...]
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug]
  telemetry:
    metrics:
      level: detailed    
    traces: [...]
    logs: [...]
```

The two batch processors create aliased metric streams, which are only
differentiated by the new component attributes. I checked that:
1. this config causes an error in the Prometheus exporter on main;
2. the error is resolved by default after applying this PR;
3. the error reappears when enabling the feature gate (this is expected)
4. scope attributes are added on the traces and logs no matter the state
of the gate.

Author: jade-guiton-dd

.chloggen/lock-attributes-gate.yaml

@@ -7,18 +7,27 @@ change_type: breaking
 component: service
 
 # A brief description of the change.  Surround your text with quotes ("") if it needs to start with a backtick (`).
-note: Set the `telemetry.newPipelineTelemetry` feature gate to stable.
+note: Restrict the `telemetry.newPipelineTelemetry` feature gate to metrics.
 
 # One or more tracking issues or pull requests related to the change
-issues: [12856]
+issues: [12856, 12933]
 
 # (Optional) One or more lines of additional information to render under the primary note.
 # These lines will be padded with 2 spaces and then inserted directly into the document.
 # Use pipe (|) for multiline entries.
 subtext: |
-  The off state of this feature gate introduced a regression, where the Collector's internal logs were missing component attributes. See issue #12870 for more details on this bug. Since there does not appear to be much benefit to disabling the gate, we are immediately moving it to stable in order to lock in the correct behavior.
+  The "off" state of this feature gate introduced a regression, where the Collector's internal logs were missing component attributes. See issue #12870 for more details on this bug.
   
-  This comes with a breaking change, where internal logs exported through OTLP will now use instrumentation scope attributes to identify the source component instead of log attributes. This does not affect the Collector's stderr output. See the changelog for v0.123.0 for a more detailed description of the gate's effects.
+  On the other hand, the "on" state introduced an issue with the Collector's default internal metrics, because the Prometheus exporter does not currently support instrumentation scope attributes.
+
+  To solve both of these issues, this change turns on the new scope attributes for logs and traces by default regardless of the feature gate.
+  However, the new scope attributes for metrics stay locked behind the feature gate, and will remain off by default until the Prometheus exporter is updated to support scope attributes.
+
+  Please understand that enabling the `telemetry.newPipelineTelemetry` feature gate may break the export of Collector metrics through, depending on your configuration.
+  Having a `batch` processor in multiple pipelines is a known trigger for this.
+  
+  This comes with a breaking change, where internal logs exported through OTLP will now use instrumentation scope attributes to identify the source component instead of log attributes.
+  This does not affect the Collector's stderr output. See the changelog for v0.123.0 for a more detailed description of the gate's effects.
 
 # Optional: The change log or logs in which this entry should be included.
 # e.g. '[user]' or '[user, api]'

internal/telemetry/telemetry.go

@@ -16,10 +16,10 @@ import (
 
 var NewPipelineTelemetryGate = featuregate.GlobalRegistry().MustRegister(
 	"telemetry.newPipelineTelemetry",
-	featuregate.StageStable,
+	featuregate.StageAlpha,
 	featuregate.WithRegisterFromVersion("v0.123.0"),
-	featuregate.WithRegisterToVersion("v0.127.0"),
 	featuregate.WithRegisterReferenceURL("https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/rfcs/component-universal-telemetry.md"),
+	featuregate.WithRegisterDescription("Injects component-identifying scope attributes in internal Collector metrics"),
 )
 
 // IMPORTANT: This struct is reexported as part of the public API of
@@ -53,6 +53,8 @@ func WithAttributeSet(ts TelemetrySettings, attrs attribute.Set) TelemetrySettin
 	ts.extraAttributes = attrs
 	ts.Logger = componentattribute.ZapLoggerWithAttributes(ts.Logger, ts.extraAttributes)
 	ts.TracerProvider = componentattribute.TracerProviderWithAttributes(ts.TracerProvider, ts.extraAttributes)
-	ts.MeterProvider = componentattribute.MeterProviderWithAttributes(ts.MeterProvider, ts.extraAttributes)
+	if NewPipelineTelemetryGate.IsEnabled() {
+		ts.MeterProvider = componentattribute.MeterProviderWithAttributes(ts.MeterProvider, ts.extraAttributes)
+	}
 	return ts
 }