Investigate ways to improve testing changes to the workflows

[xnyo]

Currently, testing changes in plugin-ci-workflows is not straightforward, and there aren't any automated checks in PRs to see if the workflows still function properly with the changes made in a PR.

This can be particularly useful for testing version bumps in renovate PRs: we often run a PR's branch in the testing repository manually before submitting the PR, but we don't usually do that for the automated PRs opened by renovate. For example, #273 introduced a breaking change that broke CI, but it was reported by the slo team only after the PR was merged.

Some ideas, not sure how feasible they are:

---

A) GHA workflows for testing plugin-ci-workflows

We can have some workflows (on: pull_request) in plugin-ci-workflows that:

• Clone one or more test plugin into the GHA runner (like https://github.com/grafana/plugins-drone-to-gha for a plugin with a backend, https://github.com/grafana/clock-panel for a frontend-only one, etc). Alternatively we could also have a folder with various testing plugins embedded in the plugin-ci-workflows repository. This can be handy for testing the various js package managers for example
• Run plugin-ci-workflows's CI for those plugins, using the branch of the current PR:
  • Run the switch references action for the current repository, so all @main references are updated to @pr-branch. Not sure if this would work? Some GHA caching may get in the way?
  • Run CI via uses: ./.github/workflows/ci.yml, so locally instead of remotely. Not sure if this would work? GHA behaves differently when referencing local vs remote workflows. I remember trying using locally checked out workflows in GHA before and it didn't work as expected
• Not sure how to confusing it would be to test CD in this way
• WIF might get in the way for GCS/publishing steps and may need to be updated. Otherwise the workflow will always fail because access is not allowed from plugin-ci-workflows non-main branches

---

B) Run CI periodically (cron) in the testing repos

We can run CI/CD periodically in the testing repos:

• https://github.com/grafana/plugins-drone-to-gha
• https://github.com/grafana/grafana-pluginsplatformprovisioned-app

And report back to Slack if it's failing. Those repos should use @main (rolling release) rather than a pinned version so we test the latest changes. However, these tests would only kick in after merging a PR, and we cannot define status checks on this. Not a huge deal since we have proper semantic versions now, we have time to test before cutting a release.

---

C) Call the workflow in the testing repos?

Maybe we can make call the CI workflow in the testing repos via the workflow_call event?

However it feels hacky/may be impossible or very hard to report the status of the run back to plugin-ci-workflows in order to define status checks for PRs

---

Maybe the sweetspot is a combination of A + C?

A) Run a smoke test for each PR in plugin-ci-workflows: just build the plugin and skip all the GCS/catalog/deployment/stuff that requires WIF, so we know that part of CI works and avoid exposing resources protected by WIF. We can then define a status check on this

C) Trigger CI in the testing repo via workflow_call as soon as a PR is merged. Here the changes should be safe since they are in main, and we don't have to wait for the cron to kick in, so we can also test the WIF-related steps (GCS, provisioned plugins deployments, etc)

[tolzhabayev]

There is no easy way to ensure all changes will always work, I agree on A+C being as much as we can feasibly afford doing. I am fine with us going down this path and learning what works well to catch actual issues and what is just overhead.

We can be more aggressive around inputs that we can use as feature toggles - e.g. enable_something: true which is false by default and let people opt-into it and then make it a default behavior. So kind of similar to how we rollout code changes in grafana.