InfluxDB decoding in sources

[jorgehermo9]

Hi, with the release of the influxDB decoding option in source decoding.codec=influxdb I thought I could use it to generate metrics from all kinds of sources, for example, with this config:

sources:
  file:
    type: exec
    command: ["cat", "sampling.txt"]
    decoding:
      codec: "influxdb"
    mode: streaming
sinks:
  console:
    type: console
    inputs: ["file"]
    encoding:
      codec: "json"

where sampling.txt are lines of valid influxDB line protocol messages.

It seems that this fails due to the fact that log sources cannot output metric types?

vector --config vector-influx.yaml --watch-config
2024-08-08T10:12:14.407424Z  INFO vector::app: Log level is enabled. level="info"
2024-08-08T10:12:14.413269Z  INFO vector::config::watcher: Creating configuration file watcher.
2024-08-08T10:12:14.416737Z  INFO vector::config::watcher: Watching configuration files.
2024-08-08T10:12:14.417237Z  INFO vector::app: Loading configs. paths=["vector-influx.yaml"]
thread 'main' panicked at lib/vector-core/src/config/mod.rs:128:9:
assertion failed: ty.contains(DataType::Log)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

If thats true and metrics cannot be generated from log sources, what are the use cases of the decoding.codec: influxdb option? Can that codec be really used in text-based sources? Or am I missing something?

For example, I would like to use a kafka source that can only output logs, but parse the messages with the influxdb codec.

I see this was discussed here #11337 (comment) but I don't see that the limitation of ty.contains(DataType::Log) for codecs is lifted in these cases, as the influxdb codec only outputs metrics and would fail for log-only output sources

I don't think it has sense to limit the output of a source based on the source type (as it is done here

vector/lib/vector-core/src/config/mod.rs

Line 127 in b2c9f27

pub fn new_logs(ty: DataType, schema_definition: schema::Definition) -> Self {

) ... It should be based on the codec, for example, if we were using decoding.codec=native_json, the Kafka source could output metrics too, and in the documentation it states that the output is exclusivelly logs output: logs

I think it would be a better fit that this was called default output: logs or so, because the "real" output depends on the codec we are using

In this case, I think any log-only output source with a metrics-only codec should be valid

[jszwedko]

Ah, yes, this should work so I'd consider it a bug.

I think you are running into this issue because influxdb is the first codec that can only emit metrics (and not logs) and so the implementation of SourceOutput doesn't have a way to model it. The other codecs that can emit metrics can also emit logs (like native_json) which is why they use ::new_logs which expects the codec to be capable of emitting logs. I think what we may need to do is to rename new_logs to new_maybe_logs and modify it so that it doesn't require the data type to include logs, but if it does, it utilizes the schema definition provided (which is only applicable to logs). Is that something you'd be interested in contributing?

And yes, the docs could benefit from updating too. The output tag is managed by whether the component documents the ability to emit a given type. For the exec source, for example, it only shows logs:

vector/website/cue/reference/components/sources/exec.cue

Lines 44 to 84 in b2c9f27

	output: logs: line: {
	description: "An individual event from exec."
	fields: {
	host: fields._local_host
	message: fields._raw_line
	timestamp: fields._current_timestamp
	data_stream: {
	common: true
	description: "The data stream from which the event originated."
	required: false
	type: string: {
	default: null
	examples: ["stdout", "stderr"]
	}
	}
	pid: {
	description: "The process ID of the command."
	required: true
	type: uint: {
	examples: [60085, 668]
	unit: null
	}
	}
	command: {
	required: true
	description: "The command that was run to generate this event."
	type: array: {
	items: type: string: {
	examples: ["echo", "Hello World!", "ls", "-la"]
	}
	}
	}
	source_type: {
	description: "The name of the source type."
	required: true
	type: string: {
	examples: ["exec"]
	}
	}
	}
	}

Which is why the documentation renders like that. I think we'd need to update any source that is capable of emitting metrics given a metrics codec to have or refactor the docs to enable indicating that a given component can emit different types depending on the codec configured.

[jorgehermo9]

I think you are running into this issue because influxdb is the first codec that can only emit metrics (and not logs) and so the implementation of SourceOutput doesn't have a way to model it. The other codecs that can emit metrics can also emit logs (like native_json) which is why they use ::new_logs which expects the codec to be capable of emitting logs. I think what we may need to do is to rename new_logs to new_maybe_logs and modify it so that it doesn't require the data type to include logs, but if it does, it utilizes the schema definition provided (which is only applicable to logs). Is that something you'd be interested in contributing?

Agree with all of that! And yes, I'd interested (although I think the parse_influxdb vrl function would fit better our use case). I'll create an issue and PR about this.

Which is why the documentation renders like that. I think we'd need to update any source that is capable of emitting metrics given a metrics codec to have or refactor the docs to enable indicating that a given component can emit different types depending on the codec configured.

That would be awesome. I think we should state this in a issue so this does not get lost. I don't how to express that, do you mind to create the issue about this? Otherwise I could try to create it by myself

[jszwedko]

Which is why the documentation renders like that. I think we'd need to update any source that is capable of emitting metrics given a metrics codec to have or refactor the docs to enable indicating that a given component can emit different types depending on the codec configured.

That would be awesome. I think we should state this in a issue so this does not get lost. I don't how to express that, do you mind to create the issue about this? Otherwise I could try to create it by myself

👍 I opened #21055 to track this documentation issue.