Multi-line with kubernetes source?

[danthegoodman1]

Things like Go, when panicking, will print multiple lines out.

Is there any way to add multi-line support to kubernetes? I see that it is available with the file source, but I don't see any settings within the k8s source that would allow multi-line

[jszwedko]

Hi @danthegoodman1 !

We've actually been moving away from having multi-line support at the source level and instead suggesting users use the reduce transform which can be used with any source to combine multi-line logs. That should work for you, but let us know if you have any issues!

[danthegoodman1]

That seems wise! Thanks

[jblang]

Doesn't using reduce this early in the pipeline limit concurrency? I don't have scientific measurements but I was trying to use reduce for multiline coalescing in conjunction with the file source, in anticipation of eventually using the kubernetes source, and when I had a large backlog of files to process, the reduce appeared to cause a bottleneck that was drastically improved by using the native multiline features in the file source.

[jblang]

@jblang just curious, what Kubernetes version are you running that Vector doesn't support?

We have multiple k8s clusters we need to use with Vector. Depending on the cluster, we have 1.18, 1.19 and 1.20.

[jszwedko]

Ah, gotcha, thanks @jblang . We do test down to 1.19 (

vector/.github/workflows/k8s_e2e.yml

Lines 107 to 113 in 6584d58

	const kubernetes_version = [
	{ version: "v1.23.3", is_essential: true },
	{ version: "v1.22.5", is_essential: true },
	{ version: "v1.21.8", is_essential: true },
	{ version: "v1.20.14", is_essential: true },
	{ version: "v1.19.8" },
	]

). I would have thought 1.18 would work too but it sounds like not :)

[jblang]

To be honest I haven't personally verified whether it works with 1.18 or not. I was going by the note in the docs stating that 1.19 is required. I do know that several APIs changed from v1alpha1 to v1beta1 around this time and we have had problems talking to our 1.18 clusters with kubectl newer than 1.19. Anyway we have plans to upgrade our 1.18 clusters soon so hopefully this will not be an issue for us for too long.

[spencergilbert]

I'm pretty sure we should work with v1.18 - I'll dig for evidence and make the change if it's the case, but I think the "requirement" is probably still down towards 1.12/1.13 since I don't think any of the API's we call have changed since then.

[jblang]

reduce is single-threaded at the moment. We do hope to improve that in the future by partitioning events along group_by fields to run multiple reduce transforms concurrently.

I suspected that this was the case based on the CPU utilization I was seeing. It would be great to have it parallelize by the groups in the future.