Hey, for the context, we currently have a workflow with many enrichment who call HTTP APIs. When an API doesn't answer, the document goes to a retry Kafka topic. When this document is replayed, the workflow will replay only the failed enrichment but not the others. In our case, some enrichments were ignored because they depended on the failed one. Consequently, several documents could have missing enrichments.
By reading the documentation, it's seems to be an expected behavior. We added an apply field to the meta_path who contain a list of all skipped and failed enrichments but it's doesn't works as expected. This field is just before the workflow but Benthos push the data inside the previous field and it's doesn't seems to care about the content.
So, is it possible to considering replay all enrichments listed in meta_path.skipped? (or maybe add an option to perform it)
Thanks!
Hey, for the context, we currently have a workflow with many enrichment who call HTTP APIs. When an API doesn't answer, the document goes to a retry Kafka topic. When this document is replayed, the workflow will replay only the failed enrichment but not the others. In our case, some enrichments were ignored because they depended on the failed one. Consequently, several documents could have missing enrichments.
By reading the documentation, it's seems to be an expected behavior. We added an
applyfield to themeta_pathwho contain a list of all skipped and failed enrichments but it's doesn't works as expected. This field is just before the workflow but Benthos push the data inside thepreviousfield and it's doesn't seems to care about the content.So, is it possible to considering replay all enrichments listed in meta_path.skipped? (or maybe add an option to perform it)
Thanks!