UDP receiver on high scale

[haimrubinstein]

Hi,

I'm new to otel so if I'm missing anything please let me know.

I have a scenario where I must use the UDP receiver on a large scale.
I've created a custom pipeline on my computer UDP -> batch processor -> file exporter.
I have nop encoding and no multiline.

I did some testing and found that per core I get 6-7K EPS, which is far from what I need.
Then I created a simple Python UDP client and I achieved 30-40K EPS.

I figured there is some code on the UDP receiver that is consuming CPU and therefore the performance is low.
After some code digging I found this code:

scanner := bufio.NewScanner(bytes.NewReader(message))
scanner.Buffer(buf, MaxUDPSize)
scanner.Split(u.splitFunc)
for scanner.Scan()

in my case, since I'm with nop encoding this is the split function the code creates

func SplitNone(maxLogSize int) bufio.SplitFunc {
	return func(data []byte, atEOF bool) (advance int, token []byte, err error) {
		if len(data) >= maxLogSize {
			return maxLogSize, data[:maxLogSize], nil
		}

		if !atEOF {
			return 0, nil, nil
		}

		if len(data) == 0 {
			return 0, nil, nil
		}
		return len(data), data, nil
	}
}

I have no need for a split, but this scanner gets created anyway.
when I removed the scanner and used the 'SplitNone' logic I got a significant! improvement to 30K EPS from the same pipeline.

Am I missing something? is there a reason to create the scanner even if there is no need for it?

before creating the scanner we can have something like this

if a.Encoding == nopEncoding {
_, b := Splitnone(message, false)
entry, _ := u.NewEntry(string(b))
u.Write(ctx, entry)
continue
}

[haimrubinstein]

@djaglowski @sumo-drosiek
can you maybe help me please?

[djaglowski]

@haim6678, thanks for raising this. It seems reasonable to me that there should be a way to disable the scanner when it's not needed, perhaps always when encoding is nop. I'd certainly welcome a PR if you're willing to draft the change.

[djaglowski]

It's more of a developer convenience that anything else, as it tokenizes logs. The multiline settings are used when a single log that extends through multiple lines, but the scanner is still useful for separating multiple lines into individual logs.

[haimrubinstein]

Maybe I'm missing something, I get that when you have a log that you want to separate (by any pattern, not just a new line) to multiple individual logs then you will want to use the scanner and the built-in split function.

But, if you have a log encoded as UTF-8 for example (and you want to keep it as a single log, even if it's spread across multiline), do you still need the scanner?

can't you just do as followed:

u.encoding.Decode(logString)
verifyMaxLogSize(logString)
entry, err := u.NewEntry(string(decoded))
u.Write(ctx, entry)

[djaglowski]

Sure, if you know for certain that you will not be tokenizing the data, then that works. This is not an assumption the component makes though, as some users will send multiple logs per request.

That said, if there's a need or major performance benefit to skipping the scanner, I'm open to enabling it. I think we need to first identify exactly how the config will support this, as the default should stay with the current behavior.

[haimrubinstein]

sure, I agree. thank you very much for your understanding.
do you want me to open a PR or do you rather do it yourself?

[djaglowski]

If you're willing, I'd happily review a PR from you. Thank you!