Plugins V2 Feedback

[Jeffail]

Hey everyone I'm in the process of implementing a brand new plugins API, you can find the API docs here: https://pkg.go.dev/github.com/Jeffail/benthos/v3@master/public/x/service

And there's an example directory showing what it looks like to use: https://github.com/benthosdev/benthos-plugin-example/tree/plugins-v2

And here's a stream of me talking through some of the functionality: https://youtu.be/x23HwcENXWM

Feel free to use this discussion as a dumping ground for thoughts, opinions, ideas and concerns. I want to know what's missing that you rely on in the old APIs, which parts don't look user friendly, etc.

[ilyaglow]

It would be nice to see an example of plugin v2 that uses metrics and logger provided explicitly as args before in register function and I suspect hidden inside service.Resources for v2. I tried to dig a bit there, but it looks like only logger is publicly exposed at the moment.

[Jeffail]

Hey @ilyaglow, I'm aiming to eventually allow access to a metrics interface but that might come later. One thing that I ought to document somewhere is that for all of the component plugin types there will already be basic metrics and tracing events done for you. I'm hoping that because of that I can get away with adding custom metrics as a nice-to-have, but I can escalate it if needed.

[ilyaglow]

Basic metrics will definitely cover the most common cases. I use custom metrics for a couple pretty huge plugins that does different kind of stuff and it's very handy that it can be easily instrumented with metrics and I can expose them from the same benthos metrics endpoint. But it's not a big problem for me to just add my own endpoint in plugin's code, it's something I can live with.

[iluminae]

We also have metrics from our processors that would only be known to our business logic - just as another datapoint.

[Jeffail]

Interesting, thanks for the feedback. This is clearly a common requirement so I've added a component to *Resources for custom metrics. I'll update the examples branch tomorrow to showcase what it looks like.

[ilyaglow]

Talking about thoughts and ideas I really miss something like TransactionChan() for processors :) It would be awesome if the new processor plugin API could at least assume and eventually support streaming interfaces in the future like io.Reader/io.Writer or some kind of channel or iterator to process very large message in chunks or produce them without a need to store the whole thing in memory.

[Jeffail]

👍 I'm thinking of adding plugin APIs for creating input and output codecs (https://www.benthos.dev/docs/components/inputs/aws_s3#codec) which would support this, although I might need to put it on the backlog for now.

[Jeffail]

The plugins v2 branch has been merged into master, but still within a package marked experimental. Main advantage here is that you'll be able to see the plugin API in https://pkg.go.dev/github.com/Jeffail/benthos/v3@master/public.

[Jeffail]

The new API is now in its permanent home at ./public/service. It's not "finished" in that it'll grow over time but I'll now be treating the existing APIs as stable and therefore subject to the same backwards compatibility guarantees as other public packages. Leaving this discussion open.

[theRealWardo]

TLDR; I believe that the plugin API has a problem that boils down to: plugin implementors struggle to make the right decision on if a connection error is the right thing to return or just a regular error. Meanwhile, operators struggle to configure error scenarios to their liking. I also believe this may be addressable with a standard set of input configuration in the plugins API.

I think the API could use some improvements in regards to error scenarios that are inherent in the current "special errors" described at ~56 minutes into your walk through. I actually am facing a similar issue to #1210 with redis_list and I believe the root cause to be the output plugin API design.

When a plugin is returning ErrNotConnected for Write operations and when an upstream fallback or similar plugin is expecting a response, I think plugin developers and Benthos users face a tough choice tuning reconnections + errors + timeouts. This is possibly best illustrated through an example config + story of how it works today:

http:
  address: 0.0.0.0:4195
  enabled: true
  root_path: /
  debug_endpoints: false

input:
  http_server:
    address: ''
    allowed_verbs:
      - POST
    path: /events
    sync_response:
      headers:
        Content-Type: application/octet-stream
      status: '200'
    timeout: 10s

output:
  broker:
    pattern: fan_out
    outputs:
      - label: error_output
        fallback:
          - label: redis_output
            redis_list:
              key: 'simple:queue'
              max_in_flight: 100
              url: redis://unreachable-redis:6379
          - label: good_file
            file:
              path: /somewhere/writeable
      - label: kafka_output
        drop_on:
          error: true
          back_pressure: 8s
          output:
              label: kafka_events
              kafka:
                addresses:
                  - unreachable-kafka:9092
                topic: good-events
                timeout: 2s
                max_retries: 1

The intent here can be described as:

• Receive HTTP requests and fan out incoming data to Redis + Kafka.
• If Redis is unavailable, write to a file.
• If Kafka is unavailable, ignore it and carry on.

This approach to handling error scenarios in practice does not work as one might think because redis_list enters into the connectLoop and the fallback to file never occurs. Changes were needed to the redis_list plugin such that it returned plain errors on write instead of special cased connection errors such that the fallback could actually occur.

The API problems:

1. Timeouts, Back Pressure, and Retires (oh my!) - the above example has 10 seconds that all of the output processors/plugins have to run, configured in the HTTP server. To avoid a timeout, the operator tunes the output timeouts to ensure retries for various outputs are hit and an error is returned before the 10 second sync response timeout to ensure the client knows "hey I accomplished what you wanted" and therefore does not retry. I've seen many a great engineer slap retries on retries. The default behavior of some plugins in Benthos is to simply try forever and just pile up CPU + RAM (redis) while others have configurable timeouts that enable excellent operator control (kafka). Consider adding a standard set of timeout and retry parameters to the plugin APIs.
2. Connectivity Errors are Errors (sometimes?) - Tuning timeouts and retries and what not could indeed tune the how connectivity errors are handled for some use cases. I think it is surprising behavior to a client awaiting a synchronous response that a connectivity issue holds up the pipeline differently than a say permissions error. In stream processing of course, maybe halting or having back pressure with an alert is what operators desire. That is to say, the plugin API should have a standard expression for configuring connectivity errors to adhere to the above timeout/retry config or a separate policy. An example of a "separate policy" would be something like "if you cannot connect, fatal error, terminate." Another thing to consider here would be a standard circuit breaker pattern which the API design today could be easily supportive of.
3. Never give up. Never surrender. - And last but not least the opposite of terminating, consider never give up as a default and warn operators appropriately of the implications in a production guide. I've used a lot of open source software that has a list of gotchas for operators and I think this is a good time to mention these changes probably come with a more detailed "Running Benthos in Production" guide possibly with some examples from the community of what we've seen in practice as well as general recommendations.

Hope this was helpful and thanks for a great tool!