Event Specific logs/Metrics

[hemanthofficial3009]

Hello,
I would like to know if anyone had tried generating or collecting the event specific information as part of the logs/metrics for an vector instance.
Scenario:
We have a vector instance that generates events (sources-Http post messages) and forwards them to another vector instance over specific network port. The messages that are received at the vector instance are then redirected/written to a specific file using the sinks plugin.

I was wondering if we can have the background information such as timestamps, received messages length and some more information that is specific to a single event at the receiver vector instance.

[jszwedko]

Hi @hemanthofficial3009 ! Sources typically add some metadata; however, the vector source does not add any metadata as it just transparently passes the event through. Are you using the vector sink/source to forward between the Vector instances?

Could you list the sources you are using and what metadata you want for each source? We could use that information to open enhancement tickets to add additional metadata where missing.

[hemanthofficial3009]

hi @jszwedko , we are using the vector sink/source to forward the messages from the client and receive the same on the collector instances of vector.
Currently we are using http, socket sources for now, probably we will need to get in more as per the requirement that come up in further development.
We are trying to fetch the

• time stamp for each message/event sent out from the source
• count of messages/events sent by the Source vector

please let me know if the input provided is clear enough or not.

thanks for responding back.

[hemanthofficial3009]

@jszwedko Thanks for responding back. Could you please provide the specific reference to the issue around tracking event latency, as the link provided earlier seems to be pointing to this discussion itself.

[jszwedko]

Oops, sorry about that. #3445 is the link.

[hemanthofficial3009]

Yeah the issue #3445 it matches exactly to what we are expecting.

[hemanthofficial3009]

@jszwedko Thanks for all the help and support, could you please let me know what is the estimated time frame by which the issue #3445 is addressed. As this feature has become a key element as part of our development.
Also I would like to know whether the vector acting as a source would add any timestamp by itself to the event or would it directly forward the event/log coming in from various sources as is?

[jszwedko]

Hi @hemanthofficial3009 ! Apologies for the delay. That currently isn't on our roadmap, but I could see us possibly digging into it next year. The vector source just passes through the event as-is, without any modification.