CI doesn't test final built extension packages

[DomAyre]

Describe the bug

This is more a lack of test coverage than a bug in the final product. But the fact CI only tests the local checkout of the code using azdev instead of installing a built package of the extension means we are actually performing zero testing on the code we ship to customers.

You can really easily expose this issue by making an extension rely on files which are not included in the built python wheel. For example, I found this issue by adding code in a new directory without the __init__.py file required to pickup that directory in building the package, CI passes with full coverage but the final released package fails with any invocation even --help.

Implementing this coverage may be difficult as currently use azdev test which as far as I know by design pulls in all of the extension source including things which may not be included in releases, meaning tests might rely on unreleased code/data.

Related command

Test coverage issue so irrelevant

Errors

Test coverage issue so irrelevant

Issue script & Debug output

Test coverage issue so irrelevant

Expected behavior

Behaviour tested in CI should represent the user experience using the extension with regular az cli not azdev

Environment Summary

Test coverage issue so irrelevant

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[DomAyre]

I've spotted another related issue. Despite our testing running against the checked out source code, they can accidentally rely on side effects of running setup.py. For example, the confcom tooling organises some code in a way that the tests rely on to pass

[DomAyre]

There are two PRs to help address this issue:

• Test built extension wheels in CI #9420
  • This does basic testing for all extensions in this repo
• [confcom] Run testing against built extension #9429
  • This properly runs all testing against the wheel but for now is only done for my extension (confcom)

If all goes well, I would recommend moving the conftest.py file defined in #9429 to the repo root so that all extensions can use it (this is subject to proper testing on all extensions)