Use vector as a library and access vector via programmable API

[niyue]

Hi there,

I am working on a C++ data project, and is evaluating if I can use vector to connect to multiple different sinks. In my case, users may submit data export jobs and I would like to feed the exported data entries into vector and send them to sinks like kafka/clickhouse/etc.

Since the sinks information are submitted by users during runtime, I need to add sink entry dynamically during runtime. This makes using toml configuration inconvenient. It seems currently the GraphQL API is read only, and I am not able to use it to add new sink during runtime.

I wonder if there is any possibility that I can use vector as a library, so that I may use tools like cbingen to create a C++ binding for Rust and use it in my project. Additionally, I think if vector can be accessed via API, more granular control/response is possible, for example, I would like to know if a data entry is really successfully delivered to the sink.

I read the documentation but find no mention about this kind of usage, is there any one who can shed some light on this and how I can give it a shot if it can be achieved (maybe requiring some hacking for the vector codebase)? Thanks.

[leebenson]

We have an open issue for considering Vector as a library, but it's nothing something that's currently on the roadmap. Hearing about potential use-cases for it is useful, and will help us evaluate whether it's something we can prioritise exploring further.

As you rightly mentioned, the API today is read-only. We're exploring driving configuration programmatically (potentially by extending the GraphQL API to take configuration at runtime), but this introduces a few design considerations that we've not yet addressed -- such as persisting a new config across restarts.

Both use-cases are very much in the early idea/discussion stages, and we don't (yet) have definitive timelines on when they may appear on our roadmap. The best I can suggest right now is to watch this space -- and in the meantime, I'd encourage users who may benefit from these features to voice their interest and let us know about potential use-cases!

[niyue]

Thanks for the prompt reply. I will keep an eye on the development.