Generic sensor (unclassified) of IPMI are always exported, generating a large number of meaningless metrics

[SilenceAdele]

Background

• My ipmi-exporter runs in native IPMI mode with --native-ipm

- "--native-ipmi"
- "--web.listen-address=:9290"

• In the current implementation, ipmi-exporter always exports generic IPMI sensors as generic metrics:

ipmi_sensor_state
ipmi_sensor_value

1. The total number of IPMI-related metrics collected for a single node during one ipmi-exporter scrape.（Total ipmi metrics: 317)

2. The ipmi_sensor_state and ipmi_sensor_value metrics account for approximately 85%（136 *2 / 317） of all IPMI metrics.

3. A large number of generic IPMI sensors are reported with NaN values in ipmi_sensor_state and zero values in ipmi_sensor_value.

ipmi-exporter-metircs.txt

Summary

• In the ipmi-exporter, all sensors except generic sensor are typed sensors with clear physical meanings, such as:

  • Fan speed (RPM / percentage)
  • Temperature（degrees C）
  • Current（Amps）
  • Voltage（Volts）
  • Power (Watts)

  These typed sensors are always exported and produce well-defined, meaningful metrics.

• Generic sensor is a fallback mechanism, used only when a sensor cannot be classified, for example:

  • Unknown or unsupported units
  • Discrete sensor types
  • Vendor-specific or proprietary sensors
  • Many generic sensors
    • state are represented as IPMI bitmasks, require OEM interpretation（NAN）
    • no runtime value (defalut: 0)

• In production environments, typed IPMI sensor（Fan speed (RPM / percentage) /Temperature（degrees C）/ Current（Amps）/ Voltage（Volts） / Power (Watts)）monitoring already provides sufficient coverage for most hardware health and alerting scenarios.

• The demand for monitoring generic IPMI sensors is very limited, while exporting them produces a large number of low-value or meaningless metrics.

• The current implementation is not elegant and introduces unnecessary performance overhead for ipmi-exporter.

• For Prometheus, in large-scale production environments, the excessive number of metrics leads to high cardinality issues, increasing ingestion, storage, and query costs.

Solutions

Introduce a configuration for --config.file to control whether generic (unclassified) IPMI sensors are exported.

The collector_gensensor_ipmi configuration is true by default to preserve backward compatibility, but users are explicitly encouraged to false in production environments to avoid exporting low-value metrics and unnecessary overhead.

The --config.file such as:

modules:
  default:
    collector_gensensor_ipmi: false

@SuperQ @bitfehler Please consider whether introducing this feature is necessary.

[SilenceAdele]

#313

The PR has been submitted. I’m happy to discuss if there are any other considerations or design concerns. @SuperQ @bitfehler

[bitfehler]

Honestly, I do not think this is a good idea at all.

• Generic sensor is a fallback mechanism, used only when a sensor cannot be classified, for example:

It's not a fallback, some sensors simply are generic.

• Unknown or unsupported units
• Discrete sensor types

Like these ^ ones.

• Vendor-specific or proprietary sensors

• Many generic sensors

  • state are represented as IPMI bitmasks, require OEM interpretation（NAN）
  • no runtime value (defalut: 0)

Now, note that you are using the experimental native IPMI implementation, so there might be some issue due to that. For example, looking at your screenshots, your generic sensor have a value but not state. I am using FreeIPMI, and my generic sensors have a state, but no value. So there might be something mixed up in the native IPMI implementation, I will look into that, but for the rest of the argument this does not matter, you may just have to substitute value and state in some cases.

Either way, just because the sensor's current value is 0, it does not mean the sensor does not produce meaningful data.

• In production environments, typed IPMI sensor（Fan speed (RPM / percentage) /Temperature（degrees C）/ Current（Amps）/ Voltage（Volts） / Power (Watts)）monitoring already provides sufficient coverage for most hardware health and alerting scenarios.

I strongly disagree. Look at the names of your generic sensors. You have chassis intrusion, ECC errors, power supply state, and so much more that you definitely should monitor in production.

• The demand for monitoring generic IPMI sensors is very limited, while exporting them produces a large number of low-value or meaningless metrics.

No, see above. Again, if some of those sensors really do produce "meaningless" metrics, it might be due to the native collectors. However, what leads you to believe your ECC error value is "meaningless"? Why would you want to exclude that from monitoring?

• For Prometheus, in large-scale production environments, the excessive number of metrics leads to high cardinality issues, increasing ingestion, storage, and query costs.

First of all, I think the amount of data produced here is peanuts compared to monitoring a Kubernetes cluster or such. But even you really do think you need to reduce the amount of data, why not just exclude sensors by ID?

Dropping all generic sensors is way to big a hammer, definitely not something people should do, especially not in production.

I am open to continue the discussion to see if something can be done to improve things for you, but I am going to close the associated PR, as such an option would steer users into a completely wrong direction.

[SilenceAdele]

Thanks a lot for the detailed reply and for clarifying your view on generic sensors.

I’ve taken your feedback on board and adjusted my local native implementation to not drop generic sensors entirely, but to reduce obviously low‑value metrics while keeping the important ones:

• For typed sensors (temperature, fan, voltage, current, power) nothing changes: they still export both value and state.
• For generic sensors, I now:
  • For native IPMI implementation:
    • Only export ipmi_sensor_value when the sensor has a real physical unit (i.e. SensorUnitType != Unspecified).
    • For sensors without physical units (e.g., discrete sensors), the ipmi_sensor_value metric is not meaningful and should not be exported.
    • Only export ipmi_sensor_state when the mapped state is not NaN.
  • For freeipmi implementation:
    • Only export ipmi_sensor_value / ipmi_sensor_state when not NaN.

This keeps things like chassis intrusion / PSU / ECC generic sensors visible via ipmi_sensor_state. When ipmi_sensor_value is NaN, this meaningless metric will not be generated.

First of all, I think the amount of data produced here is peanuts compared to monitoring a Kubernetes cluster or such. But even you really do think you need to reduce the amount of data, why not just exclude sensors by ID?

Where I still struggle a bit with the current “just use exclude_sensor_ids” recommendation is operationally, in a larger heterogeneous server environment:

• In a big cluster with many different server models and BMC firmwares, the set of generic sensors (IDs, names, types) varies per platform.
• Maintaining a long list of exclude_sensor_ids per module/model quickly becomes hard to manage and easy to drift as hardware is added/replaced.

If this still feels too opinionated for upstream, I’m fine keeping it as a local patch, but I wanted to share this approach because it seems to strike a better balance between your goal (“don’t hide important generic sensors”) and the practical pain of managing exclude_sensor_ids at scale across mixed hardware.

@bitfehler