Enable automated Copilot reviews?

[iloveeclipse]

There is a technical possibility to enable automated Copilot reviews on new PR's, see https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/7166.

Copilot reviewer would only work for PR authors with access to Github code review and only if quota allows them.

So far I found Copilot reviews on multiple PR's in different repositories as helpful.
Also Xtext already enabled that and it works fine there.

Any objections to enable Copilot by default in all SDK related organizations / repositories?

• https://github.com/eclipse-equinox
• https://github.com/eclipse-jdt
• https://github.com/eclipse-pde
• https://github.com/eclipse-platform

[merks]

My only concern is the high volume of mail becoming even more extreme and annoying.

[HannesWell]

If desired one can filter mails and auto-delete them.
I hope the Copilot bot has a dedicated mail-address.

[mbarbero]

Copilot reviewer would only work for PR authors with access to Github code review and only if quota allows them.

There's a misunderstanding. This applies to repositories that don't have the "auto copilot review" enabled. However, for repositories with this setting enabled, it's independent of the PR author.

[mbarbero]

My mistake, I was wrong. The Eclipse Foundation did not “enable code review for users without a license.” So automated Copilot reviews are tied to whether the pull request author has an active Copilot subscription.

However, soon all Eclipse Foundation project committers will be eligible for free Copilot access (once https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/7149 is resolved).

[fedejeanne]

The author?
This changes things. This would mean that we can only let Copilot review the PRs of Eclipse Committers that have opted-in to Copilot ...

?

[mbarbero]

or from users with a copilot subscription (as it is the case right now).

[mbarbero]

Additionally, users with a Copilot subscription (or those who have opted in to access Copilot through their Eclipse Foundation committer role) should be able to request a code review on pull requests where they have permission to add reviewers. In such cases, they can simply add @copilot as a reviewer: https://docs.github.com/en/copilot/how-tos/use-copilot-agents/request-a-code-review/use-code-review#using-copilot-code-review

[laeubi]

My mistake, I was wrong. The Eclipse Foundation did not “enable code review for users without a license.” So automated Copilot reviews are tied to whether the pull request author has an active Copilot subscription.

It depends a bit, e.g. we have the bot accounts for each org. If these bots where added to the eclipse-copilot org and granted access they would have about 3000 premium requests per month included (how much is used depends on the configured model to use) and one might use a github action with a PAT to automatically let the bot request a review.

Still I think it is enough if a committer adds copilot on demand... it really depends on who is changing what and the context of the pr, so from my opinion it should be a choice of the committer if its worth it in the same way as its the choice of the committer to review this particular PR.

[laeubi]

I also don't think its useful for all PRs... why not enable it on demand where it seems helpful? Having it on on a repository with a small set of contributors and low PR frequency might work but here I also suspect it produces more harm than help.

[laeubi]

Because one have to actively watch & read & react on every PR and it would save time?

This has to be done anyways... and then we will get immediately the discussions with "but my PR is not completely ready for review"..

Also there is some chance people think they need to care about PRs if its handled automatically.

For reviewers it also simplifies work if the PR is "cleaned up" before main review can start.

I found especially for Eclipse Platform it sometimes gives strange recommendation so for me it means now one needs to review the PR and the copilot to not giving hints in the wrong direction...

[ruspl-afed]

I found especially for Eclipse Platform it sometimes gives strange recommendation

I agree that AI should not be expected to provide good recommendations on how best to fit into a codebase with a 25-year history.

And, as already stated by @laeubi , this introduces an implicit phase to PR flow without good "definition of done".

However AI opinion may be considered during pipeline check.

[iloveeclipse]

I agree that AI should not be expected to provide good recommendations on how best to fit into a codebase with a 25-year history.

Actually AI has more chances to give good recommendations on older (open source) codebases because there are plenty of examples in internet how things should be done, lot of discussions, help pages, bug reports etc.

The opposite is the case, often AI doesn't know about some Java 25 specific questions because "Java 25 is not released yet" :-)

However AI opinion may be considered during pipeline check.

So AI comments are not welcome, but AI failing the pipeline is? Why that?

[fedejeanne]

In case you want to manually audit how Copilot is doing: here you can see the PRs in which copilot was involved already (74).

I don't have an opinion regarding the usefulness, I just wanted to add some context to this discussion.

[ruspl-afed]

Actually AI has more chances to give good recommendations on older (open source) codebases

I'm afraid we are talking not about features of modern Java but rather about Eclipse-unique style of coding that may have very little weight for Copilot. Actually I have no idea regarding how good Copilot is for Eclipse SDK tasks, so this is just an opinion.

So AI comments are not welcome, but AI failing the pipeline is? Why that?

I'm not saying "failing", it may be a "warning" or "info". My idea was to give it a chance to improve the code without interrupting review flow. May be "pipeline" is not the best approach for this.

[HannesWell]

While I agree that it can be helpful to have reviews enabled, I fear a bit that it could also make contributors lazy when writing proper commit messages, because the AI summaries it anyways.
But that summary isn't captured in git and as far as I saw it just describes the changes, but not the intention and reasoning (and maybe alternatives etc.).

[ruspl-afed]

as far as I saw it just describes the changes

what else it could do? AI has no idea what the change really does, it just provides more probable words for the given context.

[HannesWell]

Yes, sure and that's all good and can help. My concern is about the human-side that AI users forget about this.
I don't think this (alone) isn't a blocker, but I assume it a potential problem in the future.

[laeubi]

as far as I saw it just describes the changes

what else it could do? AI has no idea what the change really does, it just provides more probable words for the given context.

This is not completely true. If you explicitly ask for it it can write some good commit messages, but of course you need to define what you find "good" and if you give it context like issues its often not that bad.

What I think would be much more valuable, if people would use these tools before submit a PR e.g. copilot-cli can build the PR before hands, it can help can came up with a comprehensive commit message and you can even run a code review locally since a while.

In my local copilot instructions I added the following:

- don't commit your changes unless explicitly asked for - just complete your task and suggest a short summarizing commit message to the user

The result is then something like this:

So I can C&P the commit message and maybe adjust it slightly if needed or revert unwanted changes from the code-review.

[merks]

| if people would use these tools before submit

Yes, in their own fork they can do pre-polishing before spamming the entire team of committers/observes with the details of that process. (Like I do when reviewing the N&N entries both with copilot and by manually reading all the content.)