Generic sanity check for all collections

[felixfontein]

Summary

As ansible-community/ansible-build-data#114 showed we should really check the existing collections including in the Ansible package. Some ideas:

1. Set up some nightly CI that runs some basic sanity checks (like ansible-test sanity --docker -v) on all collections. Let's specify one or two of the stable branches for that.
2. Give all collections that fail this some time to fix it. If they don't, say, in two months, let's deprecate them with planned removal from Ansible 7. (Doing this for Ansible 6 is too close IMO.)
3. From then on, warn in advance once new stable branches are added to this CI (and remove old ones so that at most 1-2 of them are active at the same time), with the same rules: if a collection fails CI and doesn't fix it in a certain amount of time (with a new release), they will be deprecated and removed in the next major version of Ansible.

What do you think?

Also it would be great to do some manual review (check whether CI is set up, look at least at some of the conditions from https://github.com/ansible-collections/overview/blob/main/collection_requirements.rst, ...), but that is very time consuming, and since we already struggle with reviewing new collections, I doubt we will find enough time to do this.

[mariolenz]

Sounds reasonable. But I have a question about this:

2. Give all collections that fail this some time to fix it. If they don't, say, in two months, let's deprecate them with planned removal from Ansible 7. (Doing this for Ansible 6 is too close IMO.)

If a collection takes, say, two and a half months to fix it, would we still remove it from Ansible 7? Or would we "un-deprecate" it? Could we even do it, I mean how would we add to the changelog: We've deprecated this collection and wanted to remove it from Ansible 7 but we've changed our minds.

[felixfontein]

I guess we can still un-deprecate it if we're happy with the result (and the explanation why it took so long). I guess this would work similarly to the document we're voting on in #90, i.e. we'd have a procedure described for this with possible conditions.

[mariolenz]

I guess we can still un-deprecate it if we're happy with the result (and the explanation why it took so long).

But how would we document this in the changelog? There's deprecated_features but not undeprecated_features or something similar, is there?

[felixfontein]

I'd probably use major_changes, since both shows up in the porting guide. We can also remove the existing changelog entry if we want to.

[Andersson007]

All the ideas in the description sound good to me
+1

[MarkusTeufelberger]

Do all collections actually have sanity tests? Especially if they are only containing roles, I'm uncertain if these are even doing much, they seem very geared towards Python code in plugins from a quick look.

[felixfontein]

@MarkusTeufelberger the ones included in Ansible are required to run (and pass) sanity checks (though it's not clear how many actually do - fortinet.fortios is an example of a collection which obviously didn't run them before release). ansible-test's sanity checks don't do much for collections that only contain roles, but I think it's still a good idea to run them (it will also be rather quick), because a) the tests also check meta/runtime.yml (which you can have without plugins/modules) and changelog fragments (in case you use them), and b) hopefully sooner or later there will be a sanity check for role argument specs.

[dmsimard]

FYI I have a WIP PR in which I provide an implementation to (optionally) run sanity tests on all the collections included the package as part of the build process: ansible-community/antsibull-build#419

[dmsimard]

I've ran a comprehensive sanity test on every collection included in 6.0.0a2 (minus pylint) and found that 67 collections (out of 98) are currently failing sanity tests, data is here: https://gist.github.com/dmsimard/8a0526917398b19410581568947ee0dc

I'm not entirely positive if part of it could be due to ansible-test somehow not picking up the ignore files and need to double check but I figured I would at least share my preliminary findings for the time being and will update them if need be.

[mariolenz]

Thanks @dmsimard, great work!

I'm having a look at the output for community.vmware. vca_fw and vca_nat fail the tests:

ERROR: Found 2 validate-modules issue(s) which need to be resolved:
ERROR: plugins/modules/vca_fw.py:0:0: no-default-for-required-parameter: DOCUMENTATION.options.fw_rules: Argument is marked as required but specifies a default. Arguments with a default should not be marked as required for dictionary value @ data['options']['fw_rules']. Got {'description': ['A list of firewall rules to be added to the gateway, Please see examples on valid entries'], 'required': True, 'default': False}
ERROR: plugins/modules/vca_nat.py:0:0: no-default-for-required-parameter: DOCUMENTATION.options.nat_rules: Argument is marked as required but specifies a default. Arguments with a default should not be marked as required for dictionary value @ data['options']['nat_rules']. Got {'description': ['A list of rules to be added to the gateway, Please see examples on valid entries'], 'required': True, 'default': False}
ERROR: The 2 sanity test(s) listed below (out of 43) failed. See error output above for details.

They're deprecated, so we didn't want to invest any more time in them. I thought this would ignore the errors:

https://github.com/ansible-collections/community.vmware/blob/2ce7a1313af74bdc96e9f476330c808f25c16033/tests/sanity/ignore-2.13.txt#L1-L12

Interestingly, vca_vapp doesn't fail the tests although we had to ignore some things there, too. That's a bit weird.

Then there are a lot of this:

DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

I wonder why we don't see this in our Zuul jobs (see here for a recent CI run).

Hmmm, we've already replaced LooseVersion from distutils.version with the one from ansible.module_utils.compat.version in ansible-collections/community.vmware#1165. Looks like we have to do the same for StrictVersion.

Nevertheless, the question remains: Why don't we see this error in the Zuul jobs?

[felixfontein]

The problems in community.general and community.network are very likely due to symlinks being converted to actual files by the build process (ansible-community/antsibull-build#218). (Also for c.g the wrong version is used, that's fixed in ansible-community/antsibull-build#421.)

[felixfontein]

@mariolenz you don't see these errors in Zuul since you don't test against stable-2.13. You only test against devel (and the errors are correctly ignored in the ignore-2.14.txt), but then you test against milestone, which is a very out of date version of stable-2.13 (for example it does not have ansible/ansible#77268 which was merged on March 17th, which is what produces these errors). It basically is the state of the devel branch on 2022-02-14.

Hmmm, we've already replaced LooseVersion from distutils.version with the one from ansible.module_utils.compat.version in ansible-collections/community.vmware#1165. Looks like we have to do the same for StrictVersion.

Since the whole of distutils is deprecated: yes, you do. LooseVersion is the most commonly used thing from distutils.version, but everything else from distutils also has to be removed. StrictVersion can be 'removed' the same way as LooseVersion.