Adds example plugin for custom ingest processor #112282

samxbr · 2024-08-28T14:01:29Z

Adds an example for creating a plugin with a simple custom ingest processor. The example processor repeats the value of an expected filed in a document, or ignores it if the expected field does not exist.

Closes #111539

Testing

Added YAML REST tests

test run: https://gradle-enterprise.elastic.co/s/iz4kges66pcai

Adds an example for creating a plugin with a simple custom ingest processor. The example processor repeats the value of an expected filed in a document, or ignores it if the expected field does not exist. Closes elastic#111539

github-actions · 2024-08-28T14:01:41Z

Documentation preview:

✨ Changed pages

joegallo · 2024-08-28T15:34:15Z

...rocessor/src/main/java/org/elasticsearch/example/customprocessor/ExampleRepeatProcessor.java

+ */
+public class ExampleRepeatProcessor extends AbstractProcessor {
+    public static final String TYPE = "repeat";
+    public static final String FILED_TO_REPEAT = "toRepeat";


s/FILED/FIELD/g

And I'd do s/toRepeat/to_repeat/g, too, we don't use a lot of camelCaseLikeThis in the Elasticsearch APIs, we mostly use snake_case_like_this.

Make sense for the snake case. By s/toRepeat/to_repeat/g do you mean using regex for the field name, is there a method available for that?

I'm using it as a shorthand for "please run this regex with your mind and do a global search and replace on this pull request".

The specific s/foo/bar/g syntax is from how I typically use sed:

$ echo "Jim was here and used toRepeat, Ted also used toRepeat." | sed -e 's/toRepeat/to_repeat/g' Jim was here and used to_repeat, Ted also used to_repeat.

Ah I see, thanks for the explanation.

joegallo · 2024-08-28T16:27:02Z

...rocessor/src/main/java/org/elasticsearch/example/customprocessor/ExampleProcessorPlugin.java

+    @Override
+    public Map<String, Processor.Factory> getProcessors(Processor.Parameters parameters) {
+        return Map.ofEntries(
+            entry(ExampleRepeatProcessor.TYPE, new ExampleRepeatProcessor.Factory())


I suspect you'd be happier with just a Map.of here. It's slightly simpler.

elasticsearchmachine · 2024-08-28T18:11:52Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine · 2024-08-28T18:13:55Z

Hi @samxbr, I've created a changelog YAML for you.

jbaiera

Had a quick idea on how we could demonstrate a little more about creating processors.

jbaiera · 2024-09-03T19:07:48Z

...rocessor/src/main/java/org/elasticsearch/example/customprocessor/ExampleRepeatProcessor.java

+ */
+public class ExampleRepeatProcessor extends AbstractProcessor {
+    public static final String TYPE = "repeat";
+    public static final String FILED_TO_REPEAT = "to_repeat";


I think FILED_TO_REPEAT still needs changed to FIELD_TO_REPEAT

Oh I missed this

jbaiera · 2024-09-03T19:13:31Z

...rocessor/src/main/java/org/elasticsearch/example/customprocessor/ExampleRepeatProcessor.java

+            String description,
+            Map<String, Object> config
+        ) {
+            return new ExampleRepeatProcessor(tag, description);


It might be worth demonstrating how a user can pass properties into the processor here. For instance, most ES processors would have you specify which field name on a document to repeat. Maybe making a change so that the processor definition accepts a configuration parameter called field in the definition that will later be used to identify which document field name to repeat instead of having a single property name that is expected by the processor.

I thought about demonstrating passing in field name, and supporting Mustache template for the field name as well. But then I thought maybe it's better to keep this to minimum required for a processor plugin to avoid confusion.

I am thinking we keep this example simple, and add a reference link to an existing processor (like Set) that demonstrates passing in the field name and other features like Mustache.

What do you think? I am open to either way.

We probably don't need to involve using mustache in the example plugin, but using some values from the config object to show how they might be organized would be helpful.

samxbr · 2024-09-04T15:25:00Z

@elasticmachine test this please

samxbr · 2024-09-04T18:25:57Z

@elasticmachine update branch

joegallo · 2024-09-04T20:44:43Z

...-processor/src/yamlRestTest/resources/rest-api-spec/test/resthandler/20_process_document.yml

+"Does not process document without field":
+  # create ingest pipeline with custom processor
+  - do:
+      ingest.put_pipeline:


I'd recommend putting this in a setup: block so that you don't have to repeat yourself. Also you should probably remove it in a teardown:.

230_change_target_index.yml has an example of those two blocks in action.

jbaiera

Looks good! Left one last comment, should be good to merge once all comments are resolved

jbaiera · 2024-09-05T14:24:40Z

...rocessor/src/main/java/org/elasticsearch/example/customprocessor/ExampleRepeatProcessor.java

+            String description,
+            Map<String, Object> config
+        ) {
+            String field = ConfigurationUtils.readStringProperty(TYPE, tag, config, "field");


I would move "field" up to the top of the class into a public static final String just to avoid magic strings in the code.

joegallo · 2024-09-05T16:58:12Z

...st/java/org/elasticsearch/example/customprocessor/ExampleProcessorClientYamlTestSuiteIT.java

+/**
+ * {@link ExampleProcessorClientYamlTestSuiteIT} executes the plugin's REST API integration tests.
+ * <p>
+ * The tests can be executed using the command: ./gradlew :example-plugins:processor:yamlRestTest


Is this comment actually true?

Nice catch, this was carelessly copied from other examples. It's probably worth fixing this for other examples as well, I will double check them and fix if necessary in another PR. Want to keep this PR self-contained.

samxbr · 2024-09-05T19:59:00Z

@elasticmachine update branch

Adds example plugin for custom ingest processor

7cdde5f

Adds an example for creating a plugin with a simple custom ingest processor. The example processor repeats the value of an expected filed in a document, or ignores it if the expected field does not exist. Closes elastic#111539

samxbr requested a review from a team as a code owner August 28, 2024 14:01

elasticsearchmachine added v8.16.0 needs:triage Requires assignment of a team area label labels Aug 28, 2024

samxbr added Team:Data Management Meta label for data/management team and removed needs:triage Requires assignment of a team area label labels Aug 28, 2024

elasticsearchmachine added needs:triage Requires assignment of a team area label and removed Team:Data Management Meta label for data/management team labels Aug 28, 2024

samxbr added Team:Data Management Meta label for data/management team >enhancement and removed needs:triage Requires assignment of a team area label v8.16.0 labels Aug 28, 2024

elasticsearchmachine added needs:triage Requires assignment of a team area label and removed Team:Data Management Meta label for data/management team labels Aug 28, 2024

samxbr requested review from jbaiera and a team August 28, 2024 14:05

samxbr added v8.16.0 Team:Data Management Meta label for data/management team and removed needs:triage Requires assignment of a team area label labels Aug 28, 2024

elasticsearchmachine added needs:triage Requires assignment of a team area label and removed Team:Data Management Meta label for data/management team labels Aug 28, 2024

samxbr marked this pull request as draft August 28, 2024 14:10

samxbr added the Team:Data Management Meta label for data/management team label Aug 28, 2024

samxbr removed the request for review from a team August 28, 2024 14:12

samxbr removed the needs:triage Requires assignment of a team area label label Aug 28, 2024

joegallo reviewed Aug 28, 2024

View reviewed changes

samxbr marked this pull request as ready for review August 28, 2024 16:23

samxbr requested a review from a team August 28, 2024 16:23

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Aug 28, 2024

joegallo reviewed Aug 28, 2024

View reviewed changes

samxbr added the :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP label Aug 28, 2024

elasticsearchmachine added Team:Data Management Meta label for data/management team and removed needs:triage Requires assignment of a team area label labels Aug 28, 2024

samxbr added 2 commits August 28, 2024 14:13

Update docs/changelog/112282.yaml

2d59eac

Address comment 1

2bffcc5

jbaiera requested changes Sep 3, 2024

View reviewed changes

Fix name

3b72cad

elasticmachine and others added 2 commits September 4, 2024 20:25

Merge branch 'main' into feature/add-example-plugin-custom-processor

82d7e81

Add custom field name

16d4bec

samxbr requested review from jbaiera and joegallo September 4, 2024 19:52

joegallo reviewed Sep 4, 2024

View reviewed changes

jbaiera approved these changes Sep 5, 2024

View reviewed changes

samxbr and others added 5 commits September 5, 2024 11:02

Add yaml setup/teardown, address comment

e24084e

Merge branch 'main' into feature/add-example-plugin-custom-processor

7ace7c6

Drop an unused import

c4d019e

Whitespace

2a6e9ac

Fix a typo

f701626

joegallo requested changes Sep 5, 2024

View reviewed changes

Fix comment and folder name

727790f

Merge branch 'main' into feature/add-example-plugin-custom-processor

52a4581

samxbr requested a review from joegallo September 6, 2024 14:26

joegallo approved these changes Sep 6, 2024

View reviewed changes

samxbr merged commit 7cd6de7 into elastic:main Sep 6, 2024
15 checks passed

Adds example plugin for custom ingest processor #112282

Adds example plugin for custom ingest processor #112282

Uh oh!

Conversation

samxbr commented Aug 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Aug 28, 2024

Uh oh!

elasticsearchmachine commented Aug 28, 2024

Uh oh!

jbaiera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samxbr commented Sep 4, 2024

Uh oh!

samxbr commented Sep 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbaiera left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samxbr commented Sep 5, 2024

Uh oh!

Uh oh!

Uh oh!

samxbr commented Aug 28, 2024 •

edited

Loading