Develop policy around LLM generated code in our packages submissions

[lwasser]

We have our first submission that acknowledges use of LLMs in the codebase development

I appreciate that it's acknowledged in the code base. I assume that many other submissions use / have used it too, but aren't acknowledging it.
JOSS discussed this topic, but didn't land on a policy (that i can see):

openjournals/joss#1297

JOSS has also faced challenges with LLM-driven submissions, which have made it easier for users to generate more code more quickly. This in part has stressed the review pipeline (humans trying to review rapidly developed packages).

I see several potential challenges to consider here:

1. LICENSES: If maintainers use MIT / BSD-3 but the LLM-generated code was trained on other licenses that are less permissive (e.g., copy left) or other, how does that work (is it legal?)
2. Human Impact: We have humans taking time to review code that is non-human-generated (in part or full)
3. Ethical: there are significant ethical issues with LLM-generated code. However, we also know that MANY people do use LLMs in their daily workflows, and it helps them. We also know that it has the potential to be dangerous for groups that are underrepresented in open source.

We should develop a policy or at least have some language around this topic. However, I want to understand people's thoughts on this topic better first.

Other relevant links:

• JOSS statement

[jedbrown]

I would note that even if the LLM is exclusively trained on MIT-licensed projects (which is true of none of the commercial services), creating a derivative work without attribution is a license violation. GitHub terms of service are intended to cover inbound=outbound, which is more formally enshrined in the Developers Certificate of Origin (DCO). I've written here about what it would mean to affirm the DCO for LLM-generated code. https://hachyderm.io/@jedbrown/114928816429127795

[JacksonBurns]

I personally don't mind reviewing LLM code. I've reviewed automatically generated tests and presumably many lines which were autocompleted by some classic algorithm. LLMs are just a souped-up version of that, IMO.

Legally Speaking (re: Licenses)

I'm not a legal scholar, so when it comes to things like this I look to what others are doing.

Two US federal court rulings have recently been handed down which, in short, say that training LLMs on copyrighted material (specifically books in this case) is not copyright infringement. Here's a write up I found helpful. An analogy is that a person reading a book about something and then authoring their own about a similar topic is clearly not copyright infringement, so neither is this (i.e., if I learn to code by reading other people's code and then writing my own, it's OK). In my opinion this ignores the fact that AI models are capable of exactly reproducing code snippets without proper attribution, but theoretically any developer might do the same thing, so this is not a problem (?).

On the other hand, another judge has ruled that material generated by AI (specifically images) are not copyrightable, as they are not generated by a human (another write up). Does this extend to code?

My impression of this is that LLM-generated code on its own is not a copyright violation, but is not copyrightable. I suspect that LLM-generated code *reviewed by a human is just like any other code.

* loaded term - this could be them reading it line by line, or just giving it a scan through like you would a trusted junior dev

Practically Speaking

From a more pragmatic point of view, LLMs are useful for generating code and people will continue to use them. I think setting a blanket rule that having any code generated by an LLM is disqualifying and will force people away and discourage community contributions to pyOpenSci packages. I like the linked JOSS policy.

I won't comment on the ethical aspects of this, since I don't know enough about it.

[sneakers-the-rat]

first main thing is a disclosure requirement: we should have a place to check a few ways LLMs may have been used

e.g.

• code generation
• documentation
• tests
• ...other?

with a textbox to explain further, prompting to describe how much and which parts of the code were generated if any of those boxes were checked.

this package notes in its readme:

This project was written and maintained by hand, while making occasional use of Perplexity. For transparency, this is acknowledged, but the project should not be considered AI‑generated.

as far as i know perplexity is not a code generation llm that can hook into an IDE like cursor, so this probably means they asked for some snippets of code to be generated in some places? so that's a good example of asking "where is the llm generated code" so reviewers can assess

1. the scope of llm generated code - e.g. i definitely will copy/paste functions from stackoverflow and put a link back to the post in a docstring, and sporadic use for utility operations is different to me than widespread code generation in core routines, and matters for a reviewer deciding whether they want to review something, and
2. location of llm generated code - if reviewers do choose to review a package with llm code, they should probably know which it is so they can pay extra attention to it/know if something is just boilerplate they can glaze over

we should also have a note in the disclosure section like "if you do not disclose LLM usage and reviewers have reason to believe that code is generated, they may choose to discontinue the review and pyos may decline to review future submissions" or something. failure to disclose would be an ethical violation and in a traditional journal that would be like failing to disclose critical details in a methods section, and you'd expect an editor to drop a paper like that.

imo we can leave it up to reviewers and editors if they want to take on llm generated packages. i wouldn't, and i don't think we should encourage it, but the scenario that feels bad to me is some newbie is only exposed to "everyone uses LLMs now," they make their first package and are excited for external review, and then we decline to review it without them having a chance to know that LLM use could be problematic or controversial.

[hamogu]

To me, the important question is the scale: "Using LLM" can range from "I typed a few prompts and that generated all the code in this project" to essentially a fancy version of autocomplete. We've had editors that autocomplete know variable names or language commands or insert templates for e.g. a for loop for literally decades and nobody thought twice about that.
In the latter case, I would recommend to threat that the code just like hand-written code.

A problem in practice is that only the authors know where on the scale from 0 to 100 % LLM-generated their code falls and there is no way to check or evaluate that [*]; I personally dislike policies that disadvantage honest submitters (by e.g. putting up higher hurdles for submissions with LLM generated code in them) with no chance of ever even noticing bad actors. I don't think it is fair to put up significant barriers for honest submitters based on ethical concerns because that strongly incentives submitters to lie to us - and I don't think it's ethical to up a structure that actively gives rewards to people who behave unethical (by simply not declaring that their code is LLM generated).

So, I suggest to

• suggest that projects have a statement about their LLM policy and
• we can add a check-box "All LLM generated code in the codebase (if any) has been reviewed by a human" and require people to check that box before we begin a review. That's unenforceable (we don't know how carefully they read the code), but a reasonable nudge, I think. At least the first review of LLM generated code should be on the maintainers, not the reviewers. (But for large projects, our reviewers won't read every line of code anyway.)

Beyond that I don't think, from a practical point, that we can treat LLM-generated code different, simply because we can't recognize it reliably as being different.
That's independent on what your ethical stand on LLMs is - if we can't tell if code is LLM generated that we just can't treat it different.

*: While some LLMs may have a certain coding style that one could recognize, automatic detection of LLM writing has been shown many times to be inaccurate and the landscape is evolving so fast that it's impossible to keep up with that.

[jonas-eschle]

I think, practically speaking, anything that a human submits, must be acknowledged to be attributed to that human, and he is responsible for it. If it's AI generated code, but it's good, then there is nothing in the way of it.

I therefore strongly support a checkbox.

Furthermore, I'd recommend to be rather strict if there is straight out BS code generated by an LLM but obviously not manually reviewed, that is, sloppy AI coded. This strongly severs the trust and therefore the review process, and maybe alltogether prevents this package from obtaining a verification (if it happened now, it could happen again in the future. We want to guarantee careful coded packages.

[EdisonAltamirano]

I don’t think we can (or should) try to stop the progress, code is now being generated faster than ever before. Some of it is sloppy, but some of it, when paired with strong testing, can actually prove to be efficient and production-ready.

As technology advances, maybe we also need to rethink the peer review process itself. Beyond the traditional human review, it could be worth exploring how LLMs might assist reviewers, catching obvious issues, suggesting tests, or checking style consistency, so that humans can focus on the higher-level aspects like design, ethics, and overall quality.

Of course, responsibility and accountability must remain with humans, but I see real potential in a hybrid approach: LLMs as supportive tools in both writing code and reviewing it, while humans remain the final gatekeepers of quality.

[jonas-eschle]

As technology advances, maybe we also need to rethink the peer review process itself.

Fully agree on this one! It progresses, LLMs will generate more code, and that's a good thing overall. And we should definitely investigate the use of LLMs for ourselves, there should be quite a few checks that could be automatized.

Of course, responsibility and accountability must remain with humans, but I see real potential in a hybrid approach: LLMs as supportive tools in both writing code and reviewing it, while humans remain the final gatekeepers of quality.

Yes, this.

[jedbrown]

Suppose a human types // sparse matrix transpose, presses tab, and gets a page of code that still has namespaces from an existing library with copyright/attribution stripped. What does it mean that "accountability remains with humans"? What if the algorithmic system obfuscates just enough that it isn't obvious to the human that it's plagiarizing an existing library (violating that project's license)? Just because many people use a product doesn't mean it isn't manufacturing plausible deniability. If the claim is merely "I don't think copyright holders will successfully sue me for infringing", that is a statement about power rather than justice or consent.

The Does v Github litigation is ongoing (summary and status). It is plagiarism (and copyright infringement/license violation) for a human to take copyrighted code and obfuscate (e.g., by changing variable names and control flow isomorphisms) while removing attribution. Given that LLM-based systems are not even capable of clean-room design, I think it's a poor argument to claim it's like a person producing original works. (Even if a future court rules otherwise, that doesn't make it right or respectful of consent, and laws may change.)

LLMs are systems that emit derivative works of unknown provenance. It doesn't matter whether it's "good enough quality" or has been reviewed by a human. If a human was found to be creating derivative works without attribution, it would be an ethical transgression and may have legal consequences. The same standard should apply to LLM-generated products.

LLMs will generate more code, and that's a good thing overall.

I disagree. It is creating technical and social debt.

[lwasser]

Ok Y'all. I see some great comments above. Based on those and to further direct this conversation, here is a list of goals that we can consider when drafting both this policy/approach and the associated thought pieces on LLMS in scientific open source.

I am going to try to align some of the comments above with the goals I list below:

---

GOAL 1: Create a thoughtful disclosure policy around authors acknowledging the use of LLMs in submissions and how they were used in our submission process.

• Provide guidance on how to acknowledge (how and where // extent to which ) LLMs were used to develop/maintain the package.

@sneakers-the-rat The scope and location of the llm generated code - ie how much of it was llm generated - links in functions to stack overflow
@hamogu I don't think it is fair to put up significant barriers for honest submitters based on ethical concerns because that strongly incentives submitters to lie to us - and I don't think it's ethical to up a structure that actively gives rewards to people who behave unethical (by simply not declaring that their code is LLM generated).

GOAL 2: Protect Peer Review Efficiency: Minimize Burden

• Minimize review burden from low-quality submissions while avoiding false accusations about code origin.
  • e.g., Reviewing “junk” code created by an LLM without critical human review and testing burdens reviewers
• Misidentifying human code as LLM code will potentially impact newcomers disproportionally (who have lower confidence and sense of belonging) and will further divide the community.

@sneakers-the-rat Statement that allows reviewers to determine if it is “junk code” – not reviewed bhy a human

GOAL 3: Create a supportive (learning) environment for newcomers in peer review:

• Make it easy and supportive for those who disclose their use. Antigoal: don’t incentivize people to not disclose (example: make it easy and acceptable) (see @hamogu comment above regarding bad actors)
• Provide a support opportunity for authors to learn more about better ways to use these tools and
• Address potential learning problems with using these tools (@sneakers-the-rat has brought this up in Slack as well)
• Provide a learning opportunity for authors to better understand ethical challenges around the use of LLMs in open science & open source (I think many of us here ca
• Raise awareness of licensing and other issues associated with LLM-generated code

@sneakers-the-rat scenario that feels bad to me is some newbie is only exposed to "everyone uses LLMs now," they make their first package and are excited for external review, and then we decline to review it without them having a chance to know that LLM use could be problematic or controversial
@JacksonBurns I think setting a blanket rule that having any code generated by an LLM is disqualifying and will force people away and discourage community contributions to pyOpenSci packages.

Still not addressed: Licensing issues (@jedbrown ) I'm honestly not sure how to address this one, given the complexity and given our shared goal of not excluding people who are using these tools with good intentions.

Some implementation suggestions to consider:

@jonas-eschle Furthermore, I'd recommend to be rather strict if there is straight out BS code generated by an LLM but obviously not manually reviewed, that is, sloppy AI coded. This strongly severs the trust and therefore the review process, and maybe alltogether prevents this package from obtaining a verification (if it happened now, it could happen again in the future. We want to guarantee careful coded packages.
@hamogu suggest that projects have a statement about their LLM policy
@hamogu (@sneakers-the-rat referred to this too) @jonas-eschle Add a check box to the template about "All LLM-generated code in the codebase being reviewed and tested by a human prior to review ....

@EdisonAltamirano I love your suggestions around reimaging peer review!! For this specific thread, let's focus on policy and education first. And we can revisit these tools as reviewers later.

Please keep feedback coming - I just tried to distill what I see above into a set of goals that can drive our conversation and become pillars for decision making.
ps - i'll be mostly offline next week so keep the convo going and i'll chime in again and summarize further when i'm back.

[jonas-eschle]

I think @jedbrown that this is in the end an issue which needs to be decided by others, not pyOpenSci. We just follow what is generally done with open source packages, there is no need to come up with custom solutions. We're not here to check license infringment on such a level.

Even if a future court rules otherwise, that doesn't make it right or respectful of consent, and laws may change.

I think this is what makes me suspect, there is more personal resentment in your answer than actual fact based arguments, could this be? :) Because, course, we follow what the courts think is right. If they find it's fine, we don't try to overrule them. If the laws change, they change, and everybody needs to adapt.
Which, again, isn't really on us to enforce.

We care about code quality, unittests, maintainability and more, including not obvious copyright infringment. That's it. As long as courts deem it fine, we should deem it fine too, no question.

[pllim]

Hello. I am just passing by. In the end, whatever process you adopt, the ending would play out however the masses want it to be. If this is volunteer-driven and a flood of AI submissions come in, human volunteers will get burned out and disappear, then what will happen to the project itself? What courts are we talking about and can they be trusted to make fair judgments?

[elliesch]

This is a really great thread, and couldn't be more timely for me!

Just some quick thoughts on the goals outlined by @lwasser here:

• Goal 1:
  • I agree with @hamogu's that it's important to allow users to disclose, otherwise people will submit anyways and not disclose. I wonder if in addition to certifying that any AI code has been checked by a human, it would also be useful to require people to certify (just by a checkbox) that they have fully disclosed any AI use, since even when allowed some people hide their usage.
  • I do really like the idea of asking submitters to identify which lines of their codes are AI generated like @sneakers-the-rat does with stack overflow, and I wonder if we could request that those lines include adjacent explanatory comments made by a human, just to demonstrate that the submitter understands what the generated code is doing.
• Goal 3: I'm very excited about this goal because overall guidelines on LLM assistance are really needed all throughout the research pipeline, and I've been thinking a lot about this in the context of teaching dev to students.
  • Would tiered guidelines based off of a submitters' familiarity/expertise be helpful? For example, I find expert coders are much more likely to catch where an LLM is being overly verbose or ineffective, and not to use entire AI blocks but snippets instead, while a coder that is just starting out is more likely to use larger blocks and not catch when an AI uses many lines to do something that could be done in one line instead. I've seen that LLM assistance often hinders novice projects even when it can appear to be helpful to the novice.

[hamogu]

I do really like the idea of asking submitters to identify which lines of their codes are AI generated like @sneakers-the-rat does with stack overflow, and I wonder if we could request that those lines include adjacent explanatory comments made by a human

I don't think that's practically possible. If I use copilot in VScode, then most of my lines will be autogenerated to some degree; in some cases it might just be to autocomplete a single word, in others if might give me the skeleton of a for-loop to fill in manually, or it might generate a 5-line function. Even if I reject the suggestion, it will continue to offer things until it gets close enough to what I want until I eventually accept a suggestion.
Do we really want people to add comments to a large fraction of their code lines saying "three words on this line where autogenerated but I checked them" and "AI made this functios, but then I changed the variable names and made the comparison greater than instead of less than because that's what's needed here"?

An "explanatory comment adjacent to each of those lines" essentially means "if you use co-pilot, you need to comment almost every line of code by hand after you write it" - which essentially means spending way more time to comment than the coding would take in the first place and thus renders any AI assistance useless. If that's what we want, it would be more fair to say "We do not accept code written with the assistance of copilot or similar LLVMs".

If you generate a whole module from a prompt and then insert that into your pre-existing code, that's a different situation, but that's not how I (or most people who work around me) use LLVMs today.