Derive array type parameter from environment variable

[pirxthepilot]

Hello,

Say I have this config:

[sources.my_source_id]
type = "kafka"
bootstrap_servers = "10.14.22.123:9092,10.14.23.332:9092"
group_id = "consumer-group-name"
key_field = "message_key"
topics = [ "foo", "bar", "baz" ]

I would like the topic values to come from an environment variable. The most obvious way is to just use a regex:

KAFKA_TOPICS="^(foo|bar|baz)"

But I wonder if it's possible to instead set an environment variable like this

KAFKA_TOPICS="foo,bar,baz"

then have vector do some sort of split to convert it to an array? Something like:

topics = split("${KAFKA_TOPICS}", ",") 

but obviously, this doesn't work.

Thanks in advance!

[jszwedko]

Hey @pirxthepilot !

Yeah, as you observed using VRL functions in config isn't currently supported, but that is a neat idea. I'd recommend the regex.

[MadsRC]

I'm trying something similar, but I can't get the recommended regex to work. Can someone share some documentation on how using regexes with environment variables is supposed to work?

Alternatively, perhaps a config snippet that shows the source config, how it refers to the environment variable and then the actual value of the environment variable?

[bruceg]

The environment variable would be usable in a topic with something like topics = [ "${KAFKA_TOPICS}" ], where KAFKA_TOPICS='^(foo|bar|baz)'. Please post the configuration you are trying to use to see if we can identify the problem.

[MadsRC]

Ah that makes sense - I was under the impression that there were some magic build into vector that would format the regex as a list. If I'm understanding it correctly, it just provides the string as any other variable, and then it would be up to the source, in this case the Kafka source, to interpret that regex as "a list of topics".

For the record, my question was related to issue 17172

[bruceg]

You are understanding it correctly. In the case of the Kafka source, it specifically looks for a leading ^ to determine if the string is a regex.