Dynamically add *.toml files at runtime

[cdancy]

Hello All,

We have a use-case where vector would be running in a k8s cluster and we'd like to update the vector instance with a new toml file under certain circumstances. By adding additional files to the /etc/vector directory will they be magically picked up and used or would we need to edit the /etc/vector/vector.toml file? Does the playground/graphql API support adding additional files or would that be something we needed to code up to get additional entries added in (again, assuming vector would pick those up anyway)?

The general idea is to be able to dynamically add new transforms and sinks at runtime as needs arise and how in the world can we do that in a sane way. Any ideas one way or another?

[jszwedko]

Hi @cdancy !

You can use --config-dir to specify a directory of configuration files to load. You can then also use --watch-config to have Vector to pick up changes to files in that directory and reload the configuration.

[cdancy]

@jszwedko thanks. Can we use graphql to add additional files/entries/whatever dynamically? There is not a lot of documentation from the vector side of things so I'm not sure if this is a supported thing or not. Any idea or feel for what the community at large is doing for things like this potentially in a k8s context?

[jszwedko]

Hey!

No, the GraphQL API is read-only at this point. The only way to add/remove components at this point is to modify the configuration files and reload or restart Vector.

[cdancy]

@jszwedko yeah just tried copying a file into the container to see if that would work but no go due to it being a bare bones container.

Is there any open issue/whatever to follow for this feature?

[jszwedko]

I'm actually don't know that we have an issue specifically tracking this sort of feature. Would you want to open one and describe your use case for it in a bit more detail there?

I think, in general, Vector users in Kubernetes are making modifications by modifying the config map and then updating the pod rather than reloading live Vector instances.