Skip to content

Make to behave all unsupported processors in the same way. #269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Make to behave all unsupported processors in the same way. #269

wants to merge 1 commit into from

Conversation

mashhurs
Copy link
Collaborator

@mashhurs mashhurs commented Feb 3, 2025 •
edited
Loading

Description

Removes set_security_user processor which silently succeeds without telling its unsupported processor.
The plugin behavior for unsupported processors will be same with this change.
Adds integration tests to make sure all unsupported processors behave in the same way.

Logs

  • before:

    • events with their ingest pipelines have inference or enrich processor will be tagged with _ingest_pipeline_failure and error in the LS logs
    • events with their ingest pipelines have set_security_user will be processed without issue but no data will be set to the event - this is we don't want.

  • with this change: all events will be tagged with _ingest_pipeline_failure if unsupported processors in the ingest pipeline, errors can be seen in the LS logs:

2025-02-03 13:58:58 [2025-02-03T21:58:58,169][ERROR][co.elastic.logstash.filters.elasticintegration.IngestPipelineFactory][main] failed to create ingest pipeline `logs-apache.access-1.26.0` from pipeline configuration
2025-02-03 13:58:58 org.elasticsearch.ElasticsearchParseException: No processor type exists with name [set_security_user]
2025-02-03 13:58:58     at org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:471) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:636) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:581) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:501) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:99) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.IngestPipelineFactory.create(IngestPipelineFactory.java:49) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.SimpleIngestPipelineResolver.lambda$resolve$0(SimpleIngestPipelineResolver.java:60) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at java.util.Optional.flatMap(Optional.java:289) ~[?:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.SimpleIngestPipelineResolver.resolve(SimpleIngestPipelineResolver.java:58) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.resolver.SimpleResolverCache.lambda$reload$8(SimpleResolverCache.java:137) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.resolver.SimpleResolverCache$SimpleMultiLock.lambda$withLock$0(SimpleResolverCache.java:242) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916) ~[?:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.resolver.SimpleResolverCache$SimpleMultiLock.withLock(SimpleResolverCache.java:242) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.resolver.SimpleResolverCache.reload(SimpleResolverCache.java:136) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.resolver.SimpleResolverCache$Reloader.reloadSingleEntry(SimpleResolverCache.java:332) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4709) ~[?:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.resolver.SimpleResolverCache$Reloader.reloadOnce(SimpleResolverCache.java:328) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at co.elastic.logstash.filters.elasticintegration.resolver.CacheReloadService.runOneIteration(CacheReloadService.java:66) ~[logstash-filter-elastic_integration-8.17.1.jar:?]
2025-02-03 13:58:58     at com.google.common.util.concurrent.AbstractScheduledService$ServiceDelegate$Task.run(AbstractScheduledService.java:234) ~[guava-33.1.0-jre.jar:?]
2025-02-03 13:58:58     at com.google.common.util.concurrent.Callables.lambda$threadRenaming$3(Callables.java:105) ~[guava-33.1.0-jre.jar:?]
2025-02-03 13:58:58     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
2025-02-03 13:58:58     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
2025-02-03 13:58:58     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
2025-02-03 13:58:58     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
2025-02-03 13:58:58     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
2025-02-03 13:58:58     at java.lang.Thread.run(Thread.java:1583) ~[?:?]
2025-02-03 13:59:04 [2025-02-03T21:59:04,500][WARN ][co.elastic.logstash.filters.elasticintegration.EventProcessor][main][es_integ_filter] Pipeline `logs-apache.access-1.26.0` could not be loaded

mashhurs
mashhurs commented Feb 3, 2025
View reviewed changes
src/main/java/co/elastic/logstash/filters/elasticintegration/EventProcessorBuilder.java
@@ -33,7 +32,6 @@
import org.elasticsearch.painless.PainlessPlugin;
import org.elasticsearch.painless.PainlessScriptEngine;
import org.elasticsearch.painless.spi.PainlessExtension;
import org.elasticsearch.painless.spi.Whitelist;
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not used anywhere.

@mashhurs mashhurs linked an issue Feb 3, 2025 that may be closed by this pull request
Standardize the unsupported plugins behavior or document. #261
Open
@mashhurs mashhurs requested a review from yaauie March 12, 2025 20:49
@mergify Mergify
Copy link

mergify bot commented Mar 17, 2025

This pull request does not have a backport label. Could you fix it @mashhurs? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
  • backport-8.x is the label to automatically backport to the 8.x branch.
  • If no backport is necessary, please add the backport-skip label

yaauie
yaauie reviewed Jul 10, 2025
View reviewed changes
Copy link
Member

@yaauie yaauie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will need a rebase, but I approve it in principle. Please tag me so I can rubber-stamp it after the rebase has been done and the build is green.

src/main/java/co/elastic/logstash/filters/elasticintegration/EventProcessorBuilder.java
@@ -128,7 +126,6 @@ public EventProcessorBuilder() {
org.elasticsearch.ingest.common.UriPartsProcessor.TYPE));
this.addProcessorsFromPlugin(IngestUserAgentPlugin::new);
this.addProcessorsFromPlugin(RedactPlugin::new);
this.addProcessor(SetSecurityUserProcessor.TYPE, SetSecurityUserProcessor.Factory::new);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see any code references to set_security_user in elastic/integrations, and it has never been in our published list of supported processors.

I am +1 to deleting it.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, we have unsupported processors scan automation and so far not appeared.

@mashhurs
Remove set_security_user processor configs to make all unsupported pr…
5b0b57e 
…ocessors same behavior.

# Conflicts:
#	src/main/java/co/elastic/logstash/filters/elasticintegration/ingest/SetSecurityUserProcessor.java
@mashhurs mashhurs force-pushed the remove-set-security-user-processor branch from d02c7ff to 5b0b57e Compare July 10, 2025 14:43
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

@mashhurs mashhurs requested a review from yaauie July 10, 2025 15:11
@mashhurs mashhurs mentioned this pull request Jul 10, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yaauie yaauie Awaiting requested review from yaauie

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Standardize the unsupported plugins behavior or document.
3 participants
@mashhurs @elasticmachine @yaauie