Data validation processor

[twoversionsofme]

I'm evaluating data-prepper for log ingestion to OpenSearch via the new AWS feature of OpenSearch Pipelines. The other option I had been looking into was using AWS API Gateway -> Lambda (data validation) -> OpenSearch. The important thing to me is the data validation piece. I don't see any processor plugin for data validation based on some json schema. I'm new to data-prepper and would like to know if this would be a good feature to add. I watched the presentation from @jzonthemtn and wanted to get feedback before I head down the wrong track.

[dlvenable]

@twoversionsofme , Thank you for the feedback. I think we have a couple of options here:

1. If your data validation is very specific, you can write a custom plugin. Then you can run Data Prepper with that plugin.
2. If this is something generic. Say for example, matching JSON against JSON Schema, then we could have a processor in Data Prepper to support this. Would you be interested in contributing a processor for this? I'd be happy to help you get bootstrapped on it.

[twoversionsofme]

@dlvenable , I think it would only make sense to make it generic by validating against a provided JSON schema. I'd be happy to take a stab at contributing this. I've looked at some of the other processors and reviewed the contributions and developer guides. Is there anything else that I'd need to get started?

[rokibulislaam]

What's the current status of this? Has there been any advancement or initialisation? I am interested in a similar feature, specifically, I want to develop a validator that utilises OpenSearch index mappings. I am eager to contribute to it