Better support for renamed repositories

[daveverwer]

We don’t do a very good job supporting packages that have had their repository moved to another organisation or renamed within the same organisation.

Part of this comes from how we discover redirects. Our nightly PackageList process detects redirects using HTTP status codes. When we find a redirect from GitHub, we remove the old package from the package list and add the new one. The result of this is that the SPI app sees a brand new package and a removal of an old package, we have no concept of a rename inside the SPI database itself.

There are two main reasons we would want to support this:

1. Redirection of old package pages. If foo/bar was renamed to baz/qux we should also redirect people who arrive on the old foo/bar to bar/qux.
2. Redirection of documentation. Links to documentation can live a long time on third party sites and it’d be great to not break those links if a package gets renamed

While I was thinking this through, I also had some ideas on how we might implement this. There are many ways, but we could potentially:

1. Allow a collection of owner/repo string pairs that are all the previous repository locations for the repo.
2. During analysis, gather those names and reconcile them with a “package redirects” table. During analysis, we’d need to be careful to remove all old redirect records for the package being analysed so we cope with situations where a package author removes a redirect.
3. When a request is being processed, if it’s not for a valid package in the package list (so we cope with packages being recreated in their original position) then we check the redirects table and if we find one, we redirect the full path to the new repo, coping with long documentation URLs.

The entries in spi.yml could maybe look something like this:

metadata:
  previous_repositories:
    - owner: foo
      repository: bar
    - owner: baz
      repository: qux

[finestructure]

I don't think redirects can be defined in .spi.yml, because packages could claim another package's URLs that way. I.e. what's to stop /foo/bar to declare

metadata:
  previous_repositories:
    - owner: apple
      repository: swift-nio

?

I think we'd need to discover and track redirects via the actual redirects (like we do in our nightly job).

[daveverwer]

I don't think redirects can be defined in .spi.yml, because packages could claim another package's URLs that way.

My understanding was that the check against the package list would prevent that, but maybe it doesn’t cover every edge case? Any real package in the package list would always have priority.

One weakness I didn't consider is that if a popular repository moves, multiple other repositories could claim it as their redirect. We could check redirects of the claimed URLs during analysis before inserting the record.

I think we'd need to discover and track redirects via the actual redirects (like we do in our nightly job).

This is another alternative, but it would involve us keeping (and repeatedly checking) all old package repository URLs. I like that the packages table is currently only active packages.

We could keep packages clean by having another table for deleted packages, and I agree that making it opt-in using spi.yml isn't ideal. However, I think it's preferable to taking on the task of keeping and constantly checking every old repository URL.