template_path validation for streams and template policies

[teresaromero]

What does this PR do?

Add validation for policy templates template_path.

Depending on the type of package, input or integration, the template_path for the policy_template can be found of different locations.

• In the case of input type, the template_path field under policy_template is required, so if this is empty and error is thrown. If the value exist, its looked up under /agents/input folder at the root of the package.

• In the case of integration type, the template_path is under policy_template.inputs. This field is optional at the package manifest.

  • In the case of being not-empty, validation looks for the file under /agents/input folder at the root of the package.
  • In the case of being empty, the validation looks for the assigned stream of the input. Taking the input.type as key, validation looks at all the data streams manifest looking for the stream.input name that matches the input.type. Once there is a match, the template_path of the stream is also not required.
    • In case the template_field is not empty at the stream manifest, validation checks this file exists under /agents/stream folder at the root of the stream folder (data_stream//).
    • In case the template_field is empty, the default file name stream.yml.hbs is checked.

Why is it important?

When Fleet compiles the templates of the policies and the data streams, if the file does not exist or has not been declared at the manifest, it fails. This prevents this error by validating the package before it is installed.

Checklist

• Added unit tests for the new validation functions.

• Added spec tests with new testdata packages.

• I have added test packages to test/packages that prove my change is effective.

• I have added an entry in spec/changelog.yml.

Related issues

Resolves #703

[teresaromero]

test-integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15480

[teresaromero]

test-integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15480

[teresaromero]

test-integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15480

[teresaromero]

test-integrations

[elastic-vault-github-plugin-prod]

Created or updated PR in integrations repository to test this version. Check elastic/integrations#15480

[mrodm]

So, in this case if there is not template_path defined, does fleet looks for any default input template ?
I was wondering if it should be added some other validation here or not.

[teresaromero]

from what i have investigated, when an integration has inputs within its policy_template, the template_paths that are not explicitly described there, are taken from the data_streams.

https://github.com/elastic/kibana/blob/main/x-pack/platform/plugins/shared/fleet/server/services/epm/packages/assets.ts#L12

there is some logic i don't fully understand but my guess is that from the assetMap, it takes the data stream name from the package manifest inputs and looks for the data stream associated to it. Perhaps the validation to be done at package spec should be that, this inputs need to have a data_stream associated to be able to "skip" describing the template?

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: a3b56e5

Failed CI Steps

• :linux: Tests on Linux
• :windows: Tests on Windows
• Elastic Stack 9.2.0-SNAPSHOT compliance with Spec 3.5.0
• Elastic Stack 8.19.3 compliance with Spec 3.4.2
• Elastic Stack 9.0.6 compliance with Spec 3.3.5
• Elastic Stack 8.14.0 compliance with Spec 3.1.5
• Elastic Stack 8.9.0 compliance with Spec 2.7.0
• :junit: Junit annotate

History

• 💚 Build #1518 succeeded 8ec5de0
• 💚 Build #1517 succeeded 86fec86
• 💚 Build #1505 succeeded 818eff5
• 💚 Build #1502 succeeded 7b474a7

[teresaromero]

@mrodm @nchaulet I've updated the PR taking into account how fleet handled the template_paths for integrations with inputs.

After some consideration i've realized the entry point is always the policy_template of a manifest, regardless the type of the package. Then there is the difference between inputs and integrations.

I've added the train of thought on the PR description, and updated the code to validate like so.
The test data is failing as probably the packages are not aligned with this, before changing all the packages i would like to confirm this is the way.

I've unified the validation, as i was taking into account the streams where separated from the package manifest policy.. but with this change there should be just one validation rule, as the data_streams templates should be a match with the manifest inputs ???

[teresaromero]

closing this in favour of #1000
follow up PR coming for integration packages