Logstash Ingest Pipelines - Document which processors are not supported

[roaksoax]

Document that the following processors are not supported and why are not supported.

• set_security_user
• inference
• enrich
• attachment

Ideally, we should be able to document which integrations are supported more-so than processors.

[yaauie]

I agree that what we really should provide is a positive list of which elastic/integrations are supported (although since we do not own them, keeping this in-sync is going to be tricky).

From our existing positive list of what we do support, we could fairly easily script something that validates individual integrations with some caveats:

• we support integrations whose default pipeline is wholly-comprised of supported processors whose if and on_failure clauses are also wholly-comprised of supported processors.
• the pipeline processor is supported if-and-only-if the pipeline it references is fully-supported
  • pipelines referencing {{ IngestPipeline "XYZ" }} in elastic/integrations are effectively package-local, so in PACKAGENAME/data_stream/elasticsearch/ingest_pipeline/default.yml {{IngestPipeline "third-party" }} references PACKAGENAME/data_stream/elasticsearch/ingest_pipeline/third-party.yml:

    IngestManager / EPM replaces '{{IngestPipeline "some-pipeline" }}' with "PACKAGEPREFIX-some-pipeline".

    PACKAGEPREFIX will look like: "logs-PACKAGE.DATASET-VERSION".

    We rename all pipelines before installation by prefixing them with PACKAGEPREFIX so the name of the referenced pipeline matches.
    -- issue comment in elastic/integrations#14

  • if a pipeline otherwise references a pipeline by mustache template we may not be able to recurse
• we do NOT yet support the new reroute processor from Elasticsearch 8.8 (see Logstash Ingest Pipelines - support for "reroute" processor #67)
• we support the remaining processors from IngestCommon
• we support the geoip processor

[lucabelluccini]

The script processors which make use of Processors in ES cannot also be executed...

Caused by: java.lang.IllegalArgumentException: cannot resolve symbol [Processors]

In the Integrations repo we have only 2 occurrences, which could be easily replaced with pure-Painless counterparts

[leandrojmp]

Hello, is there any update on this?

We are thinking in offloading the processing of high event rate logs from Fortigate and Palo Alto for example, but we do not know if those integrations are compatible, there is no list of compatible integrations.

We would need to check each pipeline for the integrations to see if they have the supported processors or not, there should be a list of unsupported processors and also a list of compatible Integrations.

Is there any change from 2023 to now? Which processors are not supported?

[mashhurs]

👋 @leandrojmp, apologize for not covering this but we are pretty close here.
At the moment (8.17.3, 8.18.2, 8.19.1, 9.0.2 and 9.1.1 plugin versions), plugin do not support set_security_user, inference and enrich processors.

How to check if Logstash can run the integration?
We need to check integration ingest-pipeline definitions. For example, for your case we need to check if following ingest pipelines contain unsupported processors:

• Fortigate : https://github.com/elastic/integrations/tree/main/packages/fortinet_fortigate/data_stream/log/elasticsearch/ingest_pipeline
• Palo Alto: https://github.com/elastic/integrations/tree/main/packages/panw/data_stream/panos/elasticsearch/ingest_pipeline

[leandrojmp]

Thanks @mashhurs,

So we need to check every pipeline for every integration that we want to use to see if they have those processors?

Is there any roadmap to have this in a documentation as mentioned in the second comment in this issue?