Consider whether or not to migrate bugs.python.org source code to this repo

[ncoghlan]

(Migrating this proposal from http://psf.upfronthosting.co.za/roundup/meta/issue655 )

When CPython migrated to GitHub, the bugs.python.org code stayed on hg.python.org in order to remain consistent with upstream Roundup.

It would be nice if the bugs.python.org maintenance could be split somehow such that most CPython specific changes could be managed using the same process as other core-workflow tools, even if there were some lower level issues that still required making changes to a modified version of Roundup managed in Mercurial.

[ewdurbin]

We have a handful of repositories to consider:

https://hg.python.org/tracker/roundup/

This is our fork of roundup. It should remain as up to date as possible with upstream at http://hg.code.sf.net/p/roundup/code which is also a mercurial repository. I think it's fine that it remain on hg.python.org until such a time that the upstream repo is compatible with migrating it to GitHub or similar.

https://hg.python.org/tracker/python-dev/

The templates, extensions, detectors, and modules that customize bugs.python.org. I'm 100% on board with bringing that over to GitHub!

Other hosted trackers

We still also host:

Jython's tracker

https://hg.python.org/tracker/jython/ We could similarly migrate the customizations into this repository along side the CPython tracker.

Meta tracker

https://hg.python.org/tracker/meta/ which it sounds like goes away per #1

Setuptools tracker

https://hg.python.org/tracker/setuptools/ which seems defunct in favor of the issue tracker at https://github.com/pypa/setuptools/issues. Perhaps we should archive/statcify the existing tracker?

Roundup's Tracker

Not currently source controlled. I'd like to see this migrated into this repository as well.

[ncoghlan]

Making this repo a replacement for https://hg.python.org/tracker/python-dev/ sounds excellent to me, since most of the events that core-workflow folks are going to be interested in should already exist at that level.

I filed #4 to cover doing something about the legacy setuptools issue tracker.

Running http://bugs.jython.org/ and http://issues.roundup-tracker.org/ out of this repo would also be fine by me, although I'd also ask if it might make more sense to give them their own repos (as that way the webhook triggers for CI and automated redeployments would more naturally align between the repositories and the live sites).

[ezio-melotti]

I would prefer to leave python-dev on hg.p.o at least for now. If python-dev is moved here, the other instances should be moved as well, since I often have to apply the exact same patch on all 4 of them.
Using the same vcs for both the instances and roundup would also be nice.

[ncoghlan]

While maintaining consistent workflows is definitely preferable, we already have a major inconsistency between bugs.python.org, CPython itself, and the rest of the CPython workflow automation that means we can't just send a PR for bugs.python.org the way we can for our other workflow tools, and that creates a significant barrier to contribution for the bugs.python.org instance. It also makes it significantly more difficult to set up CI for bugs.python.org, since we can't just use Travis CI the way we do for CPython and other workflow repos.

While there are some rumblings along the lines of "Well maybe we should just stop using bugs.python.org entirely, then", I personally consider that to be a more drastic solution than we actually need. Instead, the lowest impact option we currently have available to mitigate the impact of the workflow inconsistency is to shift where it occurs to be between the bugs.python.org instance repo (which would move to GitHub), and the Roundup fork (which would remain on hg.python.org unless/until Roundup itself migrated to using git, whether on GitHub, GitLab, or somewhere else).

This does have flow-on effects for the other Roundup instances that the PSF hosts, and definitely shifts the burden of the inconvenience down the stack to the folks maintaining the rolling fork, but has the major advantage that most bugs.python.org contributions will be able to flow through a simplified version of the contribution process documented in the CPython developer guide.

It will only be when folks decide they want/need to modify Roundup itself that they'll need to start looking at Mercurial and non-PR based contribution processes (and even there, we could potentially mitigate the difference by making BitBucket the master copy of the Roundup fork and accepting PRs there, and switching hg.python.org over to being a PSF-hosted mirror).

[brettcannon]

@ezio-melotti What are the 4 instances? setuptools, jython, b.p.o and what?

And are these other instances providing people-power to help support their instances? If the answer is "no" then I think it might be time to ask them to consider migrating to any of the other various free open source hosting platforms that come with an issue tracker rather than have knock-on effects of trying to support them.

[ezio-melotti]

b.p.o, meta-tracker, jython, setuptools.

AFAIK no one is working on them, but b.p.o and meta are ours and setuptool is not longer used.
Jython is the only one that should be actively used.
I've updated those instances every time I did an update on Roundup and/or there was some security issue, but other than that they don't require any particular maintenance.

[ewdurbin]

http://issues.roundup-tracker.org itself is also hosted on bugs.python.org

[ezio-melotti]

While it is hosted on the same machine, it's not part of the 4 instances we maintain, and it's currently maintained by the Roundup devs.

[brettcannon]

OK, so maybe we should reach out to the Jython devs and let them know we are no longer going to push changes to their fork for them and they will need to have someone do it on their behalf? That will bring the number of instances down to one (assuming this replaces the meta tracker).

[ezio-melotti]

I'm not sure that would be the best course of action. The other instances require so little maintenance that they are not a burden (at least to me). For instance, the Jython instance got a dozen of commits in the last 6 years, and those changesets are applied to all the instances without changes.
I think it's best if we keep all the active instances together, and we keep maintaining them together -- in fact, I also think it would be better to add the roundup-tracker instance to the others.
Moreover, I believe none of the Jython developers have access to b.p.o or even care much about what happens to their instance as long as it keeps working fine.

[brettcannon]

@ezio-melotti but will it be a burden if we start to get more people involved? It just sounded like it was a burden in your earlier comment is all. I mean if you want to take it on to backport all changes that land in b.p.o's code then I have no objection.

[ezio-melotti]

I'm not sure I understand exactly what you mean, so I'll try to summarize the situation and my position.
It's a long post and I also included several things that are OT for this issue, but I'm having a hard time following all the issues/threads/discussion that came up lately and thought it would be good to have a summary somewhere (new issues/threads can then be started elsewhere if necessary).

TL;DR imho the vcs is not what's stopping people, we need to improve the docker image so that it's easier to set up a tracker and test things and we need more people that understand Roundup and are able to review patches. Moving to github will also make my life slightly harder for the time being.

Regarding our Roundup fork:

• it would be easier to maintain/update if it's kept on hg, since upstream uses hg too;
• it should not get any changes (especially b.p.o-specific ones), since we don't want to diverge from upstream;
• if we change anything, it should be backported upstream, so it would also be better to discuss the changes with upstream first. One such change is adding the rest api;
• getting more people involved in this would be helpful, but moving to github:
  • doesn't necessarily means more people will show up to help;
  • makes things more complicated for the aforementioned reasons;

Regarding the instances:

• there are currently 4 instances we maintain (b.p.o, meta-tracker, jython, setuptools) + roundup-tracker that is hosted on the same machine and manually updated by the Roundup devs;
• setuptools and the meta-tracker could be made read-only, bringing the number of instances down to 2+1;
• the roundup-tracker instance could join the instances we maintain, bringing the total to 3;
• most of the work is done on the b.p.o instance, the others are very low maintenance, to the point that I wouldn't mind maintaining the roundup-tracker instance too;
• getting more people involved in improving the b.p.o instance would be helpful, but moving the instances to github:
  • doesn't necessarily means more people will show up to help;
  • makes things more complicated if roundup is still on hg;

Regarding the tasks and where contributors would be useful:

• the tasks are:
  • provide an easy way to run the tracker locally;
  • writing new features for the b.p.o instance;
  • writing new features for our fork of roundup;
  • reviewing and approving the features;
  • backporting roundup features upstream;
  • updating roundup and our fork;
  • updating the instances;
  • deploying changes;
• the areas where help would be useful would be:
  • reviewing and approving features (especially big ones like the rest api or anything involving security);
  • providing an easy way to run the tracker locally (the docker image is a good starting point, but should be improved/made official/documented);
  • backporting things upstream (the roundup devs have helped out with this);
  • writing new features for the b.p.o instance and our fork;
  • the other tasks are just occasional busy-work that I can handle alone;
• to make the development easier, the steps would be:
  1. improve the docker image so that everyone can easily set up a tracker and test/review changes;
  2. find people that are able to review the patches;
  3. find people that can develop new features, with an adequate number of reviewers to match;
  4. possibly automate/streamline deploying;

For the future, I think we should follow these steps:

1. wait for the upcoming release of Roundup 1.6 and update our fork;
2. update our instances and possibly apply outstanding patches (e.g. the github login);
3. review and apply the rest api patch;
4. test/fix the rest api patch and backport upstream once stable (possibly before Roundup 2.0);
5. upgrade to Roundup 2.0, switch to Python 3, possibly ditch our fork and start following main upstream releases;
6. continue to improve our instance as we see fit;

In parallel to this we can also:

• keep improving the docker image;
• decide if/when/what move to github (also depending on what Roundup decides to do) and what to do with the meta tracker and setuptools instances, but I would wait after step 2;

[brettcannon]

So I'm hearing:

1. Better Dockerfile
2. Fix stuff
3. Get off of our fork with Roundup 2.0
4. Keep fixing stuff forever

So who wants to help make step 1 happen? And are people okay with helping make that happen on hg.python.org and the current development process there? If anyone answers "yes" to the first question but "no" to the second then please speak up.