PR from dependabot can be merged before all checks have passed

[OptimumCode]

The initial discussion started here when dependabot PR was incompatible with code in one of the implementations but still was automatically merged.

The CI workflow contains automerge job that tries to merge the PR once the ci has passed. However, there are also build and pre-commit-hook jobs that can fail. And the PR should not be merged in this case.

We should find a way to make sure ALL checks are green. The protection rules for branches could be used here but the build job is optional (only if something was changed inside the implementation folder). This can cause problems in other PRs which do not modify the implementations.

The most obvious solution would be to move the build job execution (it is in images workflow) inside the ci workflow. However, there might be other ways - we should investigate them and if there is not other way, try to reorganize the images and ci workflows (probably make images callable from other workflows)

[OptimumCode]

I tried to look for an action that provides a functionality to wait for another workflow to complete but they are either not-maintained:
https://github.com/lucasssvaz/wait-on-workflow
https://github.com/willgarcia/workflow-wait-action
or some proprietary actions (that probably might not maintain any compatibility or be removed at any moment):
https://github.com/smartcontractkit/.github/blob/main/actions/wait-for-workflows/action.yml

Based on that, it looks like the option to move build execution inside ci workflow is the best one:

• it will be transparent - no magic happens behind the scenes;
• nothing will be built if there are no changes in implementations - the build workflow already checks for that inside

We can add a workflow_call trigger to the images workflow and call it from ci workflow (only for PRs).

What do you think about the steps above, @Julian?

I have one more idea about how the initial goal can be achieved but it doesn't sound very easy (something related to using labels to mark the PR ready to merge). So, I would not go with that idea further...

[Julian]

I'm not sure how much effort it's worth when the even better solution I think is moving implementations to their own separate repo (one per implementation harness) which anyways is kind of necessary to better support bowtie-trend (we currently kind of hackily support multiple versions simultaneously for that, but it would be loads easier if each implementation had its own harness repo).

But if we started moving to that, it'd mean every implementation repository would get its own dependabot setup, and then in its own CI yeah it could do build and smoke in the same workflow as you suggest.

We could combine and then separate, but I think separate and then combine is likely to be less total effort.

That's just my initial guess though (and of course help is 1000% welcome for doing this!)

[OptimumCode]

Could you please explain what is bowtie-trend? Maybe share a few links if you have any? I probably have missed something happening with the project 😅

If you think the bowtie will benefit more from implementation separation I would probably close this issue and #1834. They will be resolved once all the harnesses are moved into separate repos.

Do you have a task for extraction of implementations into separate repositories? How could I help with it?

[Julian]

It's basically supporting every version of an implementation rather than just its latest!

We don't yet support it for all implementations (it requires building images and making the harness work on all versions) but for ones we support it powers the version trend graph on the site, see e.g.:

https://bowtie.report/#/implementations/js-hyperjump
https://bowtie.report/#/implementations/python-jsonschema

and I think a few more (it was done as part of GSoC last year by a good contributor Adwait Godbole.) But we have to scale it up I think for it to be more useful (and I think yeah separating implementations out is one step there).

[Julian]

And I don't have things clearly written out unfortunately mostly because step 0 is "figure out what if anything needs to change to support implementation repositories living in a separate repository" -- at least something needs to change in how we currently build the supported list of implementations at package buildtime -- that currently relies on everything living in the repo, but would need to change to something like "query the github api for all repos in this org tagged with being an implementation repo".

A way you could help I think is perhaps doing a POC for your own implementation in a repository (move the harness, move the CI which builds the image, with the goal being pushing the image to the same place so nothing else really needs to change). And then from that I think we'd learn if we forgot anything which needs to change!