Add an E2E step to scan unsupported processors. #242
Conversation
mashhurs
changed the title
| LOGSTASH_CONTAINER_NAME = "elastic-package-stack-e2e-logstash-1" | ||
| PLUGIN_NAME = "logstash-filter-elastic_integration" | ||
|
|
||
| SUPPORTED_PROCESSORS: list = [ |
There was a problem hiding this comment.
it would be nice to codify this in a shared yaml that is consumed by the EventProcessorBuilder.
Not a blocker, though. Could you file an issue to unify it?
There was a problem hiding this comment.
This is good idea! Otherwise we need to maintain both places. Yup let me see if I can do it with change. Otherwise I will create an issue and follow up later.
There was a problem hiding this comment.
Created an issue: #249
We can utilize https://github.com/c2nes/javalang to parse the Java class and compare with the processors found from the file. I will follow up and add separately.
| curr_dir = os.getcwd() | ||
| pipeline_definition_file_path = "integrations/packages/**/data_stream/**/elasticsearch/ingest_pipeline/*.yml" | ||
| files = glob.glob(os.path.join(curr_dir, pipeline_definition_file_path)) | ||
| unsupported_processors: dict = {} # {type: file} |
There was a problem hiding this comment.
should this be a dictionary of lists, so that we can get all integrations that use an unsupported processor?
There was a problem hiding this comment.
Updated with 1c46477 commit and tested by intentionally updating some files:
Exception: Unsupported processors found: {'rename_s': ['/logstash-filter-elastic_integration/integrations/packages/1password/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml', '/logstash-filter-elastic_integration/integrations/packages/1password/data_stream/signin_attempts/elasticsearch/ingest_pipeline/default.yml'], 'json_s': ['/logstash-filter-elastic_integration/integrations/packages/1password/data_stream/audit_events/elasticsearch/ingest_pipeline/default.yml']}
| self.__download_elastic_package() | ||
| self.__make_elastic_package_global() | ||
| self.__clone_integrations_repo() | ||
| self.__scan_for_unsupported_processors() |
There was a problem hiding this comment.
🤔 so if the elastic/integrations begin using an unsupported processor, our build breaks.
I think we might end up needing a "known incompatibilities" list so that we can mark something as known and get our build back to green once we file an issue to address it.
There was a problem hiding this comment.
Yup this is the idea for the long term. As soon as we will have unsupported plugin (assuming the plugin really cannot support), we will have a white list to continue the E2E without failing. For now, for better visibility breaking CI (since it is important to know which integration will become problematic) seems better to me.
There was a problem hiding this comment.
I will comment this plan in the raising failure part of the source.
💚 Build Succeeded
History
|
|
FYI: no need to backport since E2E runs on |
3 participants
Description
We don't have yet the integrations with
elastic_integrationunsupported processors. This PR adds a script to search processors the plugin cannot run.E2E run: https://buildkite.com/elastic/logstash-filter-elastic-integration-e2e/builds/195