Proposal: store docs.json in the registry

[natefaubion]

As part of #525 should we consider storing docs.json in the registry?

• The registry pipeline is already compiling projects with an appropriate compiler. We can easily add the appropriate codegen targets.
• The registry pipeline is the sole publisher to Pursuit nowadays. It seems unnecessary to support additional authentication workflows in Pursuit.
• If we store docs metadata in the registry, pursuit largely becomes a stateless registry client in principle, even though it likely needs to build it's own index.
• We have more freedom to adjust the actual format (especially wrt metadata) since the registry controls the data end-to-end.

Questions:

• Where is this actually stored? Package metadata is stored in a single-file ldjson format, but we would want to store version specific files, likely.
• Are there format changes we need to make? Is there redundant information in the data we currently publish to Pursuit that is otherwise available in the registry?
• We may need a data migration from pursuit-backups.
• We may need some additional postback mechanism to notify Pursuit of newly published packages if we don't want to rely on a cron job that pulls the registry metadata.

[thomashoneyman]

I like this idea. A couple extra points:

The registry pipeline is the sole publisher to Pursuit nowadays. It seems unnecessary to support additional authentication workflows in Pursuit.

This is for all intents and purposes true, but you technically don’t have to use the registry and can push to Pursuit without it. I’d be happy to end that and make this statement the state of the world though.

We may need a data migration from pursuit-backups.

Yeah, Pursuit contains more packages and package versions than the registry does. It feels easiest to settle on the concept that pursuit == the registry and remove packages that fail inclusion criteria for the registry. In that case we can regenerate docs at any time.

[natefaubion]

It feels easiest to settle on the concept that pursuit == the registry and remove packages that fail inclusion criteria for the registry. In that case we can regenerate docs at any time.

I feel like some might object to this on "historical" grounds. If these historical packages don't require additional information that can only be produced by the registry pipeline, then it seems like this could be grandfathered in through the pursuit-backups repo. That is, if we have "regenerate" script, then I don't think it's a major burden to take a pursuit-backups checkout location as a parameter and do that migration.

[thomashoneyman]

I’m all for keeping historical docs around if it’s trivial to do so; but I would personally be OK dropping old package docs if not. The source code still exists for them.

[natefaubion]

I've implemented a docs.json renderer in PS, and have the following thoughts and questions:

• Do we want registry docs.json to contain everything needed to render docs, or should we pull from multiple sources? These are the most pertinent metadata sources, but I'm not sure what is available exactly in the existing registry metadata for a package and version. There's an argument to be made that embedding everything makes it much easier to consume, but it also duplicates stored information.
  • packageMeta contains the bower.json compatible manifest (name, authors, license, repo, declared dependencies, etc)
  • moduleMap contains a mapping of module names to package names
  • resolvedDependencies contains exact resolved package versions
  • reExports (should we embed the full docs or just references)
• docs.json embeds the auto-derived json for the type language in the compiler. This is clearly meant to be internal and is absolute junk. We should not be storing this in the registry. This needs a separate, simplified specification and construction.
• Source spans (both in types and in doc spans) reference the full path of the file on disk when the docs were generated. Reconstructing a repo path from this is a heuristic. We should modify spans to refer to package-relative locations since spago/registry knows the actual path.
• A few minor tweaks to the declarations, but otherwise those are fine.
• Format is currently fixed to GH-isms to simplify linking (github user, repo, version tag). What does it look like to extend this to other providers and schemes so we can preserve the same linking behavior where possible? How do we link into monorepos for a specific package version?
• What about the package README? Pursuit fetches this from github directly. This apparently exists in the registry's package tarball?

So there's really two paths to discuss:

• What we actually store for docs in the registry.
• How pursuit goes about getting and consuming that information.

[thomashoneyman]

What we actually store for docs in the registry

I think it's OK to duplicate information into the docs.json for a package since this is an artifact of compilation + any post-processing we choose to do to the file. If it's an artifact, then we can regenerate it at any time by recompiling the package and running post-processing again.

My worry for duplication would be in the case we can't regenerate a docs.json later, in which case I'd only want it to contain unique information.

Presumably we'd delete all non-essential information from the compiler docs.json file, such as the internal info you mentioned, as well as any "bower-compatible" metadata that would already be captured by the registry metadata.

How pursuit goes about getting and consuming that information.

We've so far stored data about packages in GitHub repos (metadata, index) but packages themselves in a Digital Ocean bucket. Package docs feel like a better fit there, especially if they're potentially large?

[natefaubion]

If we want to store this in a DO bucket separate from the main registry artifacts, then I think we should probably make the registry's docs.json a fully self contained format, including things like the README as well.

[thomashoneyman]

I’m on board with that. It should be no trouble for all packages which can be compiled; I’m less confident about “historical” docs, i.e. those in pursuit-backups which may not compile or even be registered, but I guess we could have a differentiation between a “legacy” package version vs. a registry-supported one, and hide the legacy ones by default on Pursuit.

[natefaubion]

That's true. If we wanted to preserve READMEs for legacy publications, we would need a legacy converter script which can fetch that information.

I'm curious about your thoughts on

Format is currently fixed to GH-isms to simplify linking (github user, repo, version tag). What does it look like to extend this to other providers and schemes so we can preserve the same linking behavior where possible? How do we link into monorepos for a specific package version?

The publishing pipeline doesn't currently allow monorepos, so it's not obvious to me what information is around in the pipeline that supports this, and it may not be possible in general if it's an otherwise unknown host. Hackage, for example, hosts all source code it links to.

[thomashoneyman]

Yeah, that's a hard question. We currently support either GitHub or an arbitrary Git web location as package sources:

registry-dev/lib/src/Location.purs

Lines 25 to 29 in 772586e

	-- | The location of a package; the registry uses this data type to decide how
	-- | to fetch source code when publishing a new package version.
	data Location
	= Git GitData
	| GitHub GitHubData

For something like a GitHub monorepo we can follow the subdir to find the package, and then links work as normal from there. But for an arbitrary Git package we don't necessarily know how to construct links for the given host. Not sure what to do about that. You could, for instance, host a Git repository on a personal website and choose whatever link scheme you want.

As an alternative, we do store the source code for all packages in the DO bucket, so potentially we could follow Hackage's approach.

[natefaubion]

Do we know what it looks like to publish a monorepo package at a specific version? Packages right now expect a rigid tag/version format (vX.X.X).

[thomashoneyman]

The registry doesn’t know or care about tags; it all is based on the Publish payload you send, which must specify the version. A monorepo package is therefore the same as any other, it’s just that it’s a shared location instead of independent repositories.

Today we restrict things, but just because of Pursuit compatibility. If pursuit no longer needs a GitHub package with a tag then the registry can drop that validation step.

[natefaubion]

Today we restrict things, but just because of Pursuit compatibility. If pursuit no longer needs a GitHub package with a tag then the registry can drop that validation step.

Right, that's basically what I'm asking. And I suppose this is maybe a little off-topic to this particular issue, but relevant to the overall effort.

• Part of this effort is so that Pursuit can support monorepos and non-GitHub repos, which is necessary for the registry to allow them.
• All the restrictions are in place are to preserve the Pursuit status quo, which excudes monorepos and non-GitHub repos.
• What does Pursuit look like supporting these features? This is relevant to reigstry docs.json since Pursuit will be the primary consumer.

Presumably a monorepo may have any tag scheme it wants for package version, or none at all. It seems like the future of the registry pipeline is not having a strict git tag requirement. If I were to define a registry docs.json format, it would need to be compatible with this world, and that would likely just be based on the registry's Location type. However, there are Pursuit features that depend on the status quo, specifically code linking, and that can't be satisfied by the current Location type, even opportunistically (ie. only non-monorepo GitHub packages) because it does not include tag or commit metadata.

It sounds like if we do want to support source linking, some sort of hosted code viewer is the only option.

[thomashoneyman]

Yeah, no strict git tag requirement, though if you publish a git package you do provide a ref:

registry-dev/lib/src/Operation.purs

Line 76 in 772586e

, ref :: String

It feels like not a huge lift to do a hosted code viewer since we already have the package source code, so I’d be on board. More predictable than trying to resolve links to potentially arbitrary locations (even if practically speaking all packages are GitHub right now)

EDIT: The source code is available for any package in a tarball in an azure bucket (via digital ocean), e.g. effect@1.0.0 is available at https://packages.registry.purescript.org/effect/1.0.0.tar.gz