Issues and LLMs

[mzumsande]

It seems that many issues that are sufficiently simple for an LLM to solve are being sniped away by users who don't seem to make an effort to understand the code or come up with a solution for themselves, but just paste the LLM output into the PR/Commits. This is particularly the case for "Good First Issues".

As a result, I think that some contributors already stopped opening issues anymore for smallish things - they will either do the thing themself, let it rot on a private to-do-list or just forget about it.

This is obviously not ideal:

• Regular users will often ignore obviously LLM-generated PRs even if the code is correct and fixes a legitimate issue because no one likes discussing with an LLM in a PR (e.g.#33072)
• If issues aren't opened anymore, minor problems that should be fixed regardless will be forgotten
• Actual new contributors searching for problems to work on in order to get familiar with the codebase will have a harder time finding problems to solve
• New users solving issues are under a general LLM suspicion even if they did the work by themselves.

I've heard a suggestion to remove the "Good First Issue" label. Maybe there could be a slightly more restricted issue tracker that isn't visible to drive-by github users who aren't interested in bitcoin core specifically but just want a merged PR somewhere for their CV.

Any other ideas how to deal with this?

[jonatack]

Good points.

Actual new contributors searching for problems to work on in order to get familiar with the codebase will have a harder time finding problems to solve

When people ask, I suggest finding things to work on by reviewing others' work. This requires the new contributors to adopt a longer-term approach and try to add value by helping people and getting to know the project better before opening PRs.

The good or bad side is that this does seem to act as a filter.

---

In the BIPs repo, with bitcoin/bips#2006, we are attempting to make a first stab at addressing the LLM issue by requiring that proposed BIP drafts be "the original work of its authors and the content is of high quality, e.g. does not waste the community's time. No content may be generated by AI/LLMs and authors must proactively disclose up-front any use of AI/LLMs."

However, it does look like many PRs there are being opened to fix little things that people appear to be finding using LLMs. It's a time drain, but sometimes the fixes are worthwhile, e.g., maybe bitcoin/bips#2016. And when they are low value but still valid and not too much work to process, we've been trying to soft-key discourage them by collecting and curating them for later in a batched bimonthly or monthly PR.

[0xB10C]

I've heard a suggestion to remove the "Good First Issue" label.

Maybe worth trying to rename it to something that a doesn't match on a simple string compare, but looks similar to humans.

"good-first-issue" instead of "good first issue"?

[kanzure]

What if we create a norm of donating LLM credits to more experienced developers, instead of donating LLM output text? Anyone can choose to do this voluntarily. Developers can freely advertise on their own sites how to donate LLM credits or money for such purposes. A girhub bot can reply with information about LLM credits preferred over LLM outputs. Just an idea!

[maflcko]

I've heard a suggestion to remove the "Good First Issue" label.

Maybe worth trying to rename it to something that a doesn't match on a simple string compare, but looks similar to humans.

"good-first-issue" instead of "good first issue"?

I think the problem is wider than just the labelled issues. For example, 33709 was not marked "good first issue". I presume the "prompt engineers" do not filter by label, but rather just ask for "Please search all open issues, find one that appears easy to fix, fix it, and submit a pull".

The problem is that the LLM fails to understand the real underlying problem and usually submits a fix that looks plausible (possibly even compiles), but effectively is missing the point. It may be possible to follow-up with "review prompts" to guide the LLM in the right direction, but this seems like a waste of time, given that those reviewers could just directly prompt the LLM themselves, if they wanted to. The only workaround I see is to keep closing pull requests by such LLM bots, because it is the burden of the pull request author to fully understand their own code (and be able to fully author it themselves).

Obviously that is going to hurt some genuine new contributors, which is unfortunate, but it may be the best solution with the repo open to contributions from any GitHub account.

[ryanofsky]

The only workaround I see is to keep closing pull requests by such LLM bots

+1. I think this is a good approach. It seems fine to close these with a polite message explaining the project automatically closes LLM-drafted PR's from new contributors, because LLMs are not yet at the point where they absorb project-specific knowledge well, and that it is usually more efficient for experienced contributors to fix problems directly than work indirectly providing feedback to less experienced contributors using new programming tools. (Or if the contributor is hoping to learn more and become involved with the project, we think they can be more successful if they start by working directly with the code and developing a deep understanding about some part of the codebase.) If a poster made a substantive case that a PR should be reopened, it could be, and then if the PR became dormant or the review wasn't going well, it could be closed again or replaced, just like any PR.

[mzumsande]

What if we create a norm of donating LLM credits to more experienced developers, instead of donating LLM output text? Anyone can choose to do this voluntarily. Developers can freely advertise on their own sites how to donate LLM credits or money for such purposes. A girhub bot can reply with information about LLM credits preferred over LLM outputs. Just an idea!

I don't think lack of access to LLMs is a problem. Many regular devs (including myself) use LLMs judiciously.
I suspect the following incentives are at play:
1.) young and unexperienced devs trying to create a github portfolio so that they can get some kind of track record to find a job in real life
2.) People trying to get a commit into core so that they get some shitcoin airdrop that is tied to FOSS commits (used to be an issue some months ago, not sure if these still exist)

Often these people aren't even bitcoiners - if you look at their contributions they'll open similar PRs in random popular github repos all over the place.

[maflcko]

There would also be the option to have a dedicated policy, like https://docs.fedoraproject.org/en-US/council/policy/ai-contribution-policy/ or (WIP) llvm/llvm-project#154441, but those policies are likely ignored by the people that would need them the most and bike-shedding them can be another time sink.