Skip to content

enh(policy): add Gen AI policy to review guide #344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+77 −31
Open

enh(policy): add Gen AI policy to review guide #344

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
68cc75a
enh(policy): add Gen AI policy to review guide
lwasser Sep 11, 2025
48ffce5
Apply suggestions from code review
lwasser Sep 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions appendices/editor-in-chief-checks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ to work on them before the review process begins.
Please check our [Python packaging guide](https://www.pyopensci.org/python-package-guide) for more information on the elements
below.

- [ ] **Installation** The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (e.g., conda-forge, bioconda).
- [ ] **Installation** The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (for example, conda-forge, bioconda).
- [ ] The package imports properly into a standard Python environment `import package`.
- [ ] **Fit** The package meets criteria for [fit](https://www.pyopensci.org/software-peer-review/about/package-scope.html#what-types-of-packages-does-pyopensci-review) and [overlap](https://www.pyopensci.org/software-peer-review/about/package-scope.html#package-overlap).
- [ ] **Documentation** The package has sufficient online documentation to allow us to evaluate the package's function and scope *without installing the package*. This includes:
- [ ] User-facing documentation that overviews how to install and start using the package.
- [ ] Short quickstart tutorials that help a user understand how to use the package and what it can do for them.
- [ ] Short, quickstart tutorials that help a user understand how to use the package and what it can do for them.
- [ ] API documentation (documentation for your code's functions, classes, methods, and attributes): this includes clearly written docstrings with variables defined using a standard docstring format.
- [ ] Core GitHub repository Files
- [ ] **README** The package has a `README.md` file with a clear explanation of what the package does, instructions on how to install it, and a link to development instructions.
Expand All @@ -28,13 +28,14 @@ NOTE: We prefer that you have development instructions in your documentation too
- [ ] **Package overlap** The package doesn't entirely overlap with the functionality of other packages that have already been submitted to pyOpenSci.
- [ ] **Archive** (JOSS only, may be post-review): The repository DOI resolves correctly.
- [ ] **Version** (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?
- [ ] The package authors have disclosed the use of Generative AI tools in the development and/or maintenance of their package.

**Optional:** Let projects know that it's a great idea for projects to have a .github repository for the project organization where they can host a commonly used LICENSE, Code of Conduct, and even a YAML file with label definitions. These items will then be automatically applied to every repository in the organization to ensure consistency (but can be customized within repos too). The [SunPy project](https://github.com/sunpy/.github/) has a great example of this.

---
- [ ] [Initial onboarding survey was filled out ](https://forms.gle/F9mou7S3jhe8DMJ16)
We appreciate each maintainer of the package filling out this survey individually. :raised_hands:
Thank you, authors, in advance for setting aside five to ten minutes to do this. It truly helps our organization. :raised_hands:
Thank you, authors, in advance, xwfor setting aside five to ten minutes to do this. It truly helps our organization. :raised_hands:
---

*******
Expand Down
10 changes: 10 additions & 0 deletions appendices/gen-ai-checklist.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
```markdown
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once we are happy with this checklist, i'll open a pr to update our submission template as well.

- [ ] Some parts of the package were created using LLMs in some way.
* Please check any of the boxes below to clarify which parts of the package were impacted by LLM use
- [ ] LLMs were used to develop code
- [ ] LLMs were used to develop documentation
- [ ] LLMs were used to develop tests
- [ ] LLMs were used to develop infrastructure (CI, automation)
Comment on lines +2 to +7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so i think checklist can be good for prompting on different things, but i do think we want to have this be a text response - "used to develop code" could be anything from "had tab autocompletion on and used it once" to "wholly generated by LLMs."

So maybe something like...

  • Generative AI was used to produce some of the material in this submission
  • If the above box was checked, please describe how generative AI was used, including
    • Which parts of the submission were generated: e.g. documentation, tests, code. In addition to a general description, please specifically indicate any substantial portions of code (classes, modules, subpackages) that were wholly or primarily generated by AI.
    • The approximate scale of the generated portions: e.g. "all of the tests were generated and then checked by a human," "small routines were generated and copied into the code."
    • How the generative AI was used: e.g. line completion, help with translation, queried separately and integrated, agentic workflow.
  • If generative AI was used, the authors affirm that all generated material has been reviewed and edited for clarity, concision, correctness, and absence of machine bias. The authors are responsible for the content of their work, and affirm that it is in a state where reviewers will not be responsible for primary editing and review of machine-generated material.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this direction, but I'm stuck on a lot of questions about this part:

  • What does "authors are responsible for the content of their work" mean? Is that meant to be a statement of provenance or merely of agreement?
  • What if the submission contains a page of code that matches verbatim a different package with an incompatible license/no attribution? Is that misconduct, something a reviewer would politely ask them to fix, or acceptable as-is if it is plausible to believe it was tab-completed instead of flagrantly copied? (In a journal context, that would involve rejection and possible reporting of misconduct to the authors' institutions.)
  • What if that code was accepted in a PR from a minor contributor who is not an author on a paper? (Super common for projects with many contributors. You might have hundreds of minor contributors, but a dozen core developers and maintainers.) Is there an expectation that maintainers practice due diligence? (This is why I think DCO-style norms are so important.)
  • If the "fix" remedy is chosen, does that whole part of the code need to be rewritten using clean-room methods or is it enough to change variable names so it no longer shows up as a verbatim match?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "authors are responsible for the content of their work" mean? Is that meant to be a statement of provenance or merely of agreement?

it's a clarification within the affirmation that "the authors have reviewed and stand behind their work." regardless of provenance, the authors are the ones that are submitting the thing to be reviewed, and they are responsible for the content of the thing being reviewed.

this is meant to address this:

What if that code was accepted in a PR from a minor contributor who is not an author on a paper?

The people that are engaging in the review process take responsibility for the material they are asking to be reviewed.

What if the submission contains a page of code that matches verbatim a different package with an incompatible license/no attribution?

I think this is important but should probably be a separate item, eg. in this comment "we currently ask authors to write something about the state of the field of neighboring packages, ... if reviewers have generated some substantial part of their package that could have conceivably been "inspired by"/copied from another existing package, ask if they have searched for related implementations, and write something short about why not use that, and if the code does appear to overlap substantially, add some attribution. ... "

it might be a pretty high bar, and i would definitely be open to disagreement on it, because i both agree that we should encourage people being responsible with provenance, but also we can't ask someone to chase down the provenance of an undefinably small chunk of code. i again think in terms of facilitating review not what i would think are optimal development practices - "if there was a whole module of generated code that probably drew from a popular package, what would a reviewer need to know, and what is a standard we could expect from an author regarding code duplication from LLM code generation"

if the "fix" remedy is chosen

i would think attribution would be the preferred remedy here, but i also don't think that needs to be prescribed in the policies and can be a matter of negotiation between reviewers, editor, and author.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Attribution is only a "fix" if the licenses are compatible.

More generally, I'm worried that we're losing a culture (admittedly very inconsistently-practiced) of due diligence and replacing it with one in which everyone has plausible deniability, which is then vulnerable to DOS attack by dishonest academic-bean-farmers (and undermines the legitimacy of the entire JOSS/pyOS/+ community). If adequate redress for plagiarism in the scholarly literature was blame-free addition of the citation, it would be way more prevalent and the job of reviewer would be way more fraught and unappealing (and biased outcomes would proliferate). The system is built on presumption of good faith, with exceptions being scarce.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a happy middle ground could be an optional text-box?
I don't think many folks are currently tracking where/when they are using LLMs in a systematic or auditable way, or I would think its rare. You may however recall wanting to auto generate documentation explaining your functions.

I think the next item on the list with a human acknowledgement also sets this expectation and accountability for submitters too.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A text-box could ask to describe (or link) the project's policies on use of LLM products and what processes/due diligence it conducts to ensure provenance, attribution, and license compatibility.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey everyone 👋🏻 I'm checking back in on this PR. There is some great discussion here, and I agree that having that checklist seems like too much and a hard to answer / track item as Lauren points out. I am interested in how people are using these tools in their development. Maybe I could add that to our presubmission survey (as an optional question).

We are not a journal, so it is not our job to focus on publication-quality work per se. And it's possible that we have reviewed packages where code was copied from Stack Overflow or another package; we wouldn't necessarily know unless someone recognized it during the review process. At the same time, JOSS could publish via our review, so they expect a high-quality review; however, JOSS has minimal direction regarding general AI right now. We have a mission to educate maintainers and support maintainers as the ecosystem evolves. As such, these check boxes also serve to raise awareness.

I also like what @sneakers-the-rat laid out above to simplify, and Jed, as always, I love the thoughtful consideration of how these tools impact peer review. Let's focus on protecting our reviewers and raising awareness of some of the issues here. I'm incorporating some of Jonny's suggestions above, along with a few other ideas below.

What do you all think about this as a modification?

- [ ] Generative AI was used to produce some of the material in this submission
- [ ] If generative AI was used in this project, the authors affirm that all generated material has been reviewed and edited for clarity, concision, correctness, and absence of machine bias. 

If you have a policy around generative AI submissions, please provide a link to it below:
* Your Link Here <~- remove this line if you don't have a link->

The authors are responsible for the content of their work, and affirm that it is in a state where reviewers will not be responsible for primary editing and review of machine-generated material.

Jonny, I am not exactly sure where that last part (I blockquoted the text above ) . Is this a way of addressing the responsibility item that Jed mentioned above, considering they likely DO have AI contributions that they don't even know about if they have a large contributor base?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think many folks are currently tracking where/when they are using LLMs in a systematic or auditable way, or I would think its rare.

yeah, agreed, i think leaving it as an open-ended thing allows for that: you might not remember exactly what was generated or not, but you can probably say "i used the llm to generate initial class skeletons in most of the modules..." or something that captures the gist of the usage.

I believe strongly that it is not overburdensome to ask people to write a few sentences on the tools they used. I think a checkbox without any further explanation isn't really all that informative since the range of what "i used llms" is so wide. If they have a policy that captures how the authors used LLMs during creation of the package, that's fine, and they can include that here. However I think that we do not accomplish the goal of allowing reviewers to know what they are reviewing with a simple checkbox and an optional link to a policy.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the checkboxes are both not enough detail and may not be possible to honestly affirm.

For any bigger project (i.e., one in which not all contributors are authors), the authors can't really claim due diligence beyond project policies and maintainer procedures. Even if you just have a dozen authors whose LLM use may have evolved over time, honestly detailing those practices is a big task. I think it makes sense to ask for authors to explain their policies and procedures, and (for large projects) what they believe contributors are doing based on the maintainers' experience reviewing and interacting with contributors.

I don't think a maintainer can honestly check the second box unless the project has clear guidelines and believes it can trust everyone to abide by those guidelines. Even as an individual sole-author, what does it mean to say content is free from bias? Bias is the natural state of technology, and that applies also to people (though "AI" automates it in various ways that the human may or may not understand). Similarly, what would it mean to affirm that no LLM-generated content constitutes plagiarism? Such questions are incredibly fraught if you engage more deeply with what plagiarism means. The practical place to audit is at the process, not retrospective assessment of the artifact. I feel like asking people who are thinking critically to make such affirmations projects a shallow understanding from pyOpenSci and encourages people to be simplistic rather than critical in their responses. (And that carries over to practices outside the context of the review.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a practical resolution is to just require a free text field that describes, to the best of the authors ability, the use of LLMs to produce the thing that is being reviewed. in the case of a big package with lots of external authors, they could just write "there are lots of external contributions and we have no idea how they were developed. we manually reviewed all those, here is a link to our contributing docs and maybe it even has a section on LLM usage in it." Then they could additionally describe the things the authors do know about like their own usage of llms. We are asking for authors to describe how the work was created so that reviewers can understand what they are reviewing. If a reviewer doesn't want to review something that doesn't have a strict llm policy (or any at all), then that's fine. If a reviewer is fine with that, that's also fine.

I think the question of auditing process is a separate question to this - e.g. like how we require having docs and tests, we could require having a contributing policy that addresses LLMs. I don't necessarily think we should require that (none of my projects have an LLM policy), just saying that it sounds like a separate question: the purpose of the statement in the submission is to allow reviewers to know what they are reviewing, and any other requirements on the content of the submission are, imo, a different thing than the disclosure requirement.

re: the second checkbox, i think a minor revision to add "... has been reviewed and edited by the authors for ..." . I would probably add a preface like "The authors affirm that they are responsible for the content of their submission, and have taken appropriate measures to minimize reviewer labor..." before the specific requirements so that it's clear what the affirmation means. again, like with the disclosure, we are not asking the authors to be experts in what plaigiarism or bias means, but give the reviewers something to point to like "hey you said you reviewed this beforehand, and yet here is a bunch of dead code/a section with extremely biased outcomes/etc. so you didn't live up to the things i believed to be true when i agreed to review"


- [ ] I acknowledge that a human (at least one member of our maintainer team) has reviewed, edited, and modified all LLM-generated code and text submitted in this review to ensure it’s not biased (if text) and the code runs (if code).
```
26 changes: 16 additions & 10 deletions how-to/author-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Curious about the general timeline for pyOpenSci reviews?
:link-type: doc
:class-card: left-aligned

Read about our peer review policies.
Read through our peer review policies before submitting a package to us.
:::
::::

Expand All @@ -86,9 +86,10 @@ Before you begin this process, [please be sure to read the review process guidel

1. Please be sure that you have time to devote to making changes to your
package. During review, you will receive feedback from an editor and two reviewers. Changes could
take time. Please consider this before submitting to us. You can read more about the timeline to make changes in our [peer review policies page](../our-process/policies).
take time. Please consider this before submitting to us. You can read more about the timeline for making changes on our [peer review policies page](../our-process/policies).
2. A diverse group of volunteer editors and reviewers leads peer review.
Please be considerate when engaging with everyone online.
3. All participants in our peer review process need to follow the [pyOpenSci Code of Conduct](https://www.pyopensci.org/handbook/CODE_OF_CONDUCT.html). Please review it before submitting to us.
```

## 1. Do you plan to continue to maintain your package?
Expand Down Expand Up @@ -128,7 +129,7 @@ emphasis on long-term software maintenance and focuses more on
publication quality and citation/credit.
```

## 2. Does Your Package Meet Packaging Requirements?
## 2. Does your package meet packaging requirements?

Before submitting your project for review with pyOpenSci, make sure that
your package meets all of the requirements listed in the editor checks (see below).
Expand All @@ -147,6 +148,11 @@ for Python packaging, including discussions of:

```

:::{important}
Given the increased use of Generative AI tools (LLMs), we have developed a disclosure policy for all packages submitted to pyOpenSci (generative-ai-policy). Please review this before submitting to us.
:::


```{hint}
**Do you have questions about Python packaging or our peer review process?**

Expand All @@ -169,7 +175,7 @@ that you can use to create a high-quality
Python package.](https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html)
```

## 3. Is Your Package in Scope for pyOpenSci?
## 3. Is your package in scope for pyOpenSci?

Next, check to see if your package falls within the topical and technical scope of pyOpenSci. If you aren't
sure about whether your package fits within pyOpenSci's scope (below), submit
Expand All @@ -187,13 +193,13 @@ Our current categories for determining package scope are below:
Click here to view our technical and domain scope requirements.
```

## 4. Submit Your Package for Peer Review
## 4. Submit your package for peer review

To submit your package for peer review, you can
open an issue in our [pyopensci/software-review repo](https://github.com/pyOpenSci/software-review/issues/new/choose/)
repository and fill out the [Submit Software for Review](https://github.com/pyOpenSci/software-submission/issues/new?template=submit-software-for-review.md) issue template.

## 5. Editor-in-Chief Reviews Package for Scope and Minimal Infrastructure Criteria
## 5. Editor-in-Chief reviews package for scope and minimal infrastructure criteria

Once the issue is opened, our editor-in-chief and an editor from our editorial board will review your submission within
**2 weeks** and respond with next steps. The editor may request that you make updates
Expand All @@ -206,7 +212,7 @@ package if it does not fall within our scope.
Click here to view the editor checks that will be used to evaluate your package.
```

## 6. The Review Begins
## 6. The review begins

If your package meets the minimal criteria for being
reviewed, it may then be given to an editor with appropriate domain experience
Expand All @@ -216,7 +222,7 @@ issue within **3 weeks**. Reviewers can also open issues in your package reposit
We prefer issues that link back to the review as they document changes made to your
package that were triggered by our review process.

## 7. Response to Reviews
## 7. Response to reviews

You should respond to reviewers’ comments within **2 weeks** of the
last-submitted review. You can make updates to your package at any time. We
Expand All @@ -230,7 +236,7 @@ Once the reviewers are happy with the changes that you've made to the package, t
editor will review everything and accept your package into the pyOpenSci ecosystem.
Congratulations! You are almost done!

## My Package is Approved, Now What?
## My package is approved, now what?

Congratulations on being accepted into the pyOpenSci community of maintainers!
Once your package is approved, a few things will happen:
Expand All @@ -245,7 +251,7 @@ Once your package is approved, a few things will happen:

If you'd like to submit your package to JOSS, you can do so now. Remember that JOSS will accept our review as theirs, so **you DO NOT need to go through another review**. Read more below.

### Journal of Open Source Software (JOSS) Submission
### Journal of Open Source Software (JOSS) submission

pyOpenSci has a [partnership with JOSS](JOSS), where our review is accepted by JOSS by
default if the package fits into the JOSS scope.
Expand Down
65 changes: 47 additions & 18 deletions our-process/policies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,15 @@
## Review process guidelines

pyOpenSci packages are reviewed for quality, fit, scope, documentation, and
usability. The review process is similar to a manuscript review, however, it
usability. The review process is similar to a manuscript review; however, it
has a stronger focus on Python packaging best practices.

Unlike a manuscript review, our peer review process is an ongoing conversation.
Once all major issues and questions are addressed, the review editor will make
a decision to accept, hold, or reject the package.
Once all major issues and questions are addressed, the review editor will decide
to accept, hold, or reject the package.

Rejections are usually done early in the process, before the review process
begins. In rare cases, a package may also not be on-boarded into the pyOpenSci
begins. In rare cases, a package may also not be onboarded into the pyOpenSci
ecosystem after review & revision.

It is ultimately the editor’s decision on whether or not to reject the package
Expand All @@ -20,26 +20,27 @@ based on how the reviews are addressed.
## Review communication approach

Communication between authors, reviewers, and editors takes
place on GitHub. You can, however choose to contact the editor by email if
place on GitHub. You can, however, choose to contact the editor by email if
needed.

When submitting a package, please make sure that your GitHub notification
settings are setup to notify you when you receive feedback on the review issue.
settings are turned on for the software-submission repository to notify you
when you receive feedback on a review issue.

## Submitting your package for review in other venues

We recommend submitting your package for review with pyOpenSci before
submitting a software paper describing the package to a journal.

Review feedback may result in major improvements and updates to your package,
including changes that could be break package functionality.
Review feedback may result in significant improvements and updates to your package,
including changes that could break package functionality.

Applying reviewer or editor recommendations to your package can improve your
users' experience with future versions of your package even if your package is
users' experience with future versions of your package, even if your package is
already published on `PyPI` or `conda-forge`.

> Please do not submit your package for review while it or an associated
> manuscript is also under review at another venue, as this may result on
> manuscript is also under review at another venue, as this may result in
> conflicting requests for changes from two sets of reviewers.

### Publication with Journal of Open Source Software (JOSS)
Expand Down Expand Up @@ -77,6 +78,33 @@ a conflict of interest if:
In the case where none of the associate editors can serve as editor, an
external guest editor will be recruited to lead the package review.

(generative-ai-policy)=
## Policy for use of generative AI / LLMs

:::{admonition} How this policy was developed
:class: important

The policy below was co-developed by the pyOpenSci community. Its goals are:

* Acknowledgment of and transparency around the widespread use of Generative AI tools (with a focus on Large Language Models (LLMs) in Open Source.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

along with this - allow reviewers to make informed decisions about what they choose to review, and allow authors to have reviewers that align with their values and practices. facilitate a review process that aligns with the ethics and values of its participants.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appreciate this. let me adjust this section

* Protect peer review efficiency: Ensure human review of any LLM-generated contributions to a package to protect editor and reviewer volunteer time in our peer review process.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a little confusing - we want to avoid pyOS reviewers being the first ones to review LLM-generated code, so by "human review" we mean "prior author review." so maybe something like "Ensure an equitable balance of labor, where authors have ensured that generated material is in a state that minimizes review time, and reviewers are not responsible for correcting errors and unclarity from machine generated code. The PyOS review process should not be used as a mechanism for outsourcing human review of generated code."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally - let me try to turn your comment into an inline edit. I agree it's confusing as written but the goal is definitely equitable balance of labor - authors need to carefully review FIRST . i like that language

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Protect peer review efficiency: Ensure human review of any LLM-generated contributions to a package to protect editor and reviewer volunteer time in our peer review process.
* Ensure an equitable balance of effort in the peer review process. Authors acknowledge that a human has carefully reviewed parts of the package that are AI-generated. Generated material should be in a state that minimizes review time. Our reviewers are not responsible for correcting errors in machine-generated content.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is another comment Jonny wrote that I want to capture, which is in the main body of this review. I copied it below

allow reviewers to make informed decisions about what they choose to review, and allow authors to have reviewers that align with their values and practices. facilitate a review process that aligns with the ethics and values of its participants.

I agree with Moritz that it is VERY hard to find reviewers right now so values alignment will be tricky. However, I think allowing a reviewer to decline reviewing on the basis of LLM-generated content is entirely acceptable. And if they look at the issue submission, they will see that LLMs are involved and how.

Should we add a statement to our reviewer guide about this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes - and in general we could remind reviewers that they are volunteering and can withdraw their review for whatever reason. this is sort of obvious by the nature of volunteer review, but just want to ensure reviewers like "you are in control of your time and we are glad to have it, but don't want to ask you to do something you are not comfortable with"

* Raise awareness of challenges that Generative AI tools present to the scientific (and broader) open source community.
[Please see this GitHub issue for a discussion of the topic.](https://github.com/pyOpenSci/software-peer-review/issues/331)
Comment on lines +91 to +92
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Raise awareness of challenges that Generative AI tools present to the scientific (and broader) open source community.
[Please see this GitHub issue for a discussion of the topic.](https://github.com/pyOpenSci/software-peer-review/issues/331)
* Disclosure further allows reviewers and editors to make conscious decisions around the types of packages that they wish to review code for.
* Raise awareness of challenges that Generative AI tools present to the scientific (and broader) open source community.
[Please see this GitHub issue for a discussion of the topic.](https://github.com/pyOpenSci/software-peer-review/issues/331)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sneakers-the-rat, please have a look at this - it may not quite be the right language. But essentially the idea here is that if a reviewer starts to review a package and they have strong opinions about LLM use in open source, then they get to a module that is clearly LLM output copy-pasta. That could be aggravating to them and it could also bias the review. So, disclosure essentially not only protects people's time, but it also helps create a more equitable and value-aligned process (I think).

:::

### Disclosure of generative AI use in pyOpenSci reviewed packages

* When you submit a package to pyOpenSci, please disclose any use of LLMs (Large Language Models) in your package’s generation by checking the appropriate boxes on our software submission form. Disclosure should generally include what parts of your package were developed using LLM tools.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

along with above, checking boxes -> describing your use of generative AI.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See what you think of my edits below.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* When you submit a package to pyOpenSci, please disclose any use of LLMs (Large Language Models) in your package’s generation by checking the appropriate boxes on our software submission form. Disclosure should generally include what parts of your package were developed using LLM tools.
* When you submit a package to pyOpenSci, please disclose any use of LLMs (Large Language Models) in your package’s generation by checking the appropriate boxes and describing your use of generative AI in it's development and/or maintenance. Disclosure should include what parts of your package were developed using Generative AI (LLMs).

* Please also disclose this use of Generative AI tools in your package's `README.md` file and in any modules where generative AI contributions have been implemented.
* We require that all aspects of your package have been reviewed carefully by a human on your maintainer team. Please ensure all text and code have been carefully checked for bias, bugs, and issues before submitting to pyOpenSci.
* Your acknowledgment of using Generative AI will not impact the success of your submission unless you have blindly copied text and code into your package without careful review and evaluation of its accuracy, and for any systemic bias.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this can realistically be promised. I can imagine some reviewers would not want to participate in such a review. Imagine if you are asked to review a paper and they acknowledge that anonymous non-consenting students and a paid ghostwriter wrote portions of the paper, but the author has read it all so it's good to go. If you believe that constitutes research misconduct because it violates the policies of your university and of other journals that you typically review for (which may use COPE and CRediT).

You could say that reviewers assigned to the submission will have to promise it won't affect their review. (IMO, reviewers need to be aware of this and have an informed opportunity for choose whether to consent.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Maybe something like "use of generative AI, if disclosed, does not disqualify your package from review. Your disclosure is used to match your submission with reviewers with similar values and experience regarding gen AI. Incomplete disclosure of gen AI could affect your review success, as reviewers are volunteers and retain discretion to discontinue their review if this, or any other part of your submission appears inaccurate on closer inspection."

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we want to match package with "reviewers who share your values". That (a) makes ii even harder to find reviewers (instead of asking anyone, I know need to ask first "Where do you stand o the copyright and LLM debate?") and (b) makes pyopensci fall into two parts the "PyOpenSci with AI friendly authors and reviewers" and the "PyOS with anti- LLM authors and reviewers".

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also "where do you stand on publication ethics". Unpaid reviewing of scholarly work is service to the epistemic integrity of a community and knowledge. I think the motivations to review for pyOS are more akin to reviewing for scholarly journals than to conducting an external security audit (which is almost always a paid activity and has a narrow purpose).

To impose on reviewers a uniform policy that violations of publication ethics norms must not influence their review will drive those reviewers away. The remaining reviewers, being "LLM-friendly", may be prone to use LLMs to create the appearance that they have conducted a review. (This is a huge problem for CS conferences where chairs are desperate for reviewers and people would like the professional recognition of being a member of the program committee without doing the work.)

Copy link
Contributor

@sneakers-the-rat sneakers-the-rat Sep 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, ya ^

We are responding to an ethical and practical problem that has been forced upon us, not creating one. If we do nothing then the submissions could be clogged with generated packages, anyone who has an ethical problem with that (selfishly, me) leaves. If we require disclosure, we allow people the opportunity akin to informed consent to choose how they contribute their time. If reviewers are fine with LLM-generated packages, great. If not, also fine, they can choose what to review. The standards of review don't speak to LLM generated code and should apply uniformly.

Similarly, from the readers PoV, if the reader is fine with LLM-generated packages, cool, that's still useful information. If the reader is not, also cool, they know what to avoid.

From the author PoV, if I am fine with LLM-Generated packages, fine, if I'm seeking external review I should feel comfortable being honest about how its development process works. If I am not fine with LLM-generated packages, I would not submit my work here without a thoughtful LLM policy because I would not want my work associated with what I believe to be unethical slop, and disclosure allows my work to remain separate.

Requiring disclosure expands the reviewer, submitter, and reader pools, not contracts them. I am not seeing an articulation of harms from having a nonuniform reviewer pool when the review standards are uniform and checked by an editor who is a third party to reviewer choice.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok this is also super helpful. I added this statement because @hamogu made a powerful and valid argument that some people will intentionally not disclose use. We could inadvertently penalize honest people with great intentions who are using these tools in productive and thoughtful ways.

I don't want to create a divide. I also don't want prejudice in our system where any package that has been supported by an LLM is rejected.

We can have opinions about these tools and still work together to make informed decisions about where to invest our time. If a reviewer finds that a module is poorly written / wrong copy pasta LLM output, they could opt to say "no thank you" or they could opt to ask the authors to rewrite, we pause the review and then they restart the review when fixes are made.

What about something like: (open to suggestions)

  • Your acknowledgment of using Generative AI will not prejudice the success of your submission. However, a reviewer can and will ask you to revisit your package's content if it appears that section have been copied and pasted from other sources without human review.

* If the review team (comprised of the editor and reviewers) determines that the code and text in the package are too challenging to review, they can decide to pause and/or discontinue the review following this policy’s guidelines.

Below is the checklist that you will need to respond to in our submission form:

```{include} ../appendices/gen-ai-checklist.md
```

## Review timelines and on-hold reviews

At any time, an author can choose to have their submission put on hold
Expand Down Expand Up @@ -114,9 +142,9 @@ including issues, pull requests, and dates of the last release and last commit
to the package repository. Activity is defined as a repository commit, pull
request, or release.

We will flag packages that haven't been updated within a 1 year/ 12 month time
We will flag packages that haven't been updated within a 1-year/12-month time
period based on activity. Packages with no activity after 12 months will be
flagged. At that time, pyOpenSci editorial team member will contact the package
flagged. At that time, a pyOpenSci editorial team member will contact the package
maintainers to evaluate the maintenance status of their package.

(archive-process)=
Expand All @@ -130,15 +158,15 @@ ecosystem.
In cases where the community heavily uses a package, we may
collaborate with the community to identify reasonable next steps, such as
assisting in finding a new maintainer. If a solution for the ongoing package
maintenance is not found, the package will be archived within the pyOpenSci
ecosystem.
maintenance of your package is not found, the package will be archived within
the pyOpenSci ecosystem. [See section on archiving below.](package-archive)

If a sub-community decides to fork and maintain the package, we are open to
working with the new maintainers to register the newly forked package within
our ecosystem. The original package will be archived with a link to the new
fork.

We will also add a note to any blogs written that highlight your tool that
We will also add a note to any blogs written that highlight your tool, that
the package is no longer maintained.

### Quality commitment
Expand All @@ -156,19 +184,20 @@ in touch with us if they do need to step down from maintaining a tool.
### Requesting Package Removal from the pyOpenSci Ecosystem

In the unlikely scenario that a contributor of a package requests removal of
their package from our ecosystem, we retain the right offer the last / most
their package from our ecosystem, we retain the right to offer the last/most
recently released version of that package in our ecosystem for archival
purposes only.

(package-archive)=
### Archiving a package

If a package appears to be no longer maintained, we will mark it as
archived which moves the package from our
archived, which moves the package from our
[main package listing](https://www.pyopensci.org/python-packages.html#all-packages)
to our [archived packaging](https://www.pyopensci.org/python-packages.html#archived-packages)
listing section.

To archive a pyOpenSci approved package, add the
To archive a pyOpenSci-approved package, add the
[archive label](https://github.com/pyOpenSci/software-submission/issues?q=label%3Aarchived)
to the original review issue. Once this label is applied to the issue, the
website will automatically update to reflect this status. If at any point
Expand Down
Loading