Bethos seems to be stuck retrying failed messages

[acziryak-everhealth]

Long story short is that while trying to get nginx logs into Kafka processed by Benthos and put into Elastic, I enabled logs that didn't conform to any of the processors in Benthos.

So when I got the logs->kafka-> benthos part working, I started seeing this logs in Benthos:

root@benthos-server:/etc/benthos# journalctl -xeu benthos-udweb -n 500000 | grep 'No matches found for payload: 2024/03/25 04:18:27' | wc -l
103
root@benthos-server:/etc/benthos# journalctl -xeu benthos-udweb -n 500000 | grep 'No matches found for payload: 2024/03/25 04:18:27' | cut -d ' ' -f 1-3
Mar 25 15:41:47
Mar 25 15:44:47
Mar 25 15:57:18
Mar 25 15:57:49
Mar 25 15:58:49
Mar 25 16:11:12
Mar 25 16:12:43
Mar 25 16:13:13
Mar 25 16:13:43
Mar 25 16:14:13
Mar 25 16:14:45
Mar 25 16:15:16
Mar 25 16:15:46
Mar 25 16:16:17
Mar 25 16:16:46
Mar 25 16:17:16
Mar 25 16:18:48
Mar 25 16:19:18
Mar 25 16:19:50
Mar 25 16:20:17
Mar 25 16:20:48
Mar 25 16:21:20
Mar 25 16:21:48
Mar 25 16:22:20
Mar 25 16:22:49
Mar 25 16:23:20
Mar 25 16:23:22
Mar 25 16:23:52
Mar 25 16:24:22
Mar 25 16:24:52
Mar 25 16:24:55
Mar 25 16:24:55
Mar 25 16:25:25
Mar 25 16:25:29
Mar 25 16:25:53
Mar 25 16:25:56
Mar 25 16:26:24
Mar 25 16:26:27
Mar 25 16:26:53
Mar 25 16:26:57
Mar 25 16:27:00
Mar 25 16:27:23
Mar 25 16:27:25
Mar 25 16:27:29
Mar 25 16:27:55
Mar 25 16:27:58
Mar 25 16:28:26
Mar 25 16:28:56
Mar 25 16:28:59
Mar 25 16:29:27
Mar 25 16:29:29
Mar 25 16:29:56
Mar 25 16:29:59
Mar 25 16:30:26
Mar 25 16:30:30
Mar 25 16:30:56
Mar 25 16:30:57
Mar 25 16:31:29
[...snip...]

So it looks like the "No matches for payload" error is causing this to get re-run often. I can't seem to skip it, and Benthos doesn't seem to be processing further messages, it's just hung there. How can I un-stick this?

EDIT: FWIW, this isn't production data, so if the answer is to blow away the topic in kafka and re-create it, that works too.

[acziryak-everhealth]

To add insult to injury, what it tells me to do in the docs: https://www.benthos.dev/docs/configuration/error_handling throws an error:

root@benthos-server:/etc/benthos# grep mapping streams/udweb.yaml
    - mapping: root = if errored() { deleted() }
root@benthos-server:/etc/benthos# benthos lint streams/udweb.yaml
streams/udweb.yaml: line 119: unable to infer processor type, candidates were: [mapping]

[mihaitodor]

Hey @acziryak 👋 That "No matches found for payload" error you're seeing is coming from here and it's only emitted when debug logging is enabled. If you're fine to skip that message when no matches are found then your approach should work, just that you likely have a formatting error in your yaml config or maybe you're using an old version of Benthos which doesn't have the mapping processor available yet, in which case you'll want to use bloblang instead. If you're still stuck on V3, please use the docs here: https://v3.benthos.dev/docs/configuration/error_handling

[acziryak-everhealth]

Yeah, that did end up being the case. Adding the bloblang processor worked, and I was able to churn through my queue.