The future of the `rust-releases` crate

[foresterre]

Goals

• Merging and combining sources: sometimes we want to combine multiple sources to come to a better release index. For example, some sources are updated quicker after a release is published than others, some are easier to fetch, but are also incomplete with respect to data they contain. The merged index should have a more complete view than the individual parts.
  • Consider different merging strategies; we don't always have data with timestamps, and not always is later also more correct.
• A built in source of truth index for releases in the past, baked statically into a crate (this is mostly useful if you can merge it with newer data)
  • A daily ran workflow which updates the crate automatically
  • Consider how to deal with corrections made by humans
• More metadata: currently we only cared about versions, and let dependencies gather additional metadata themselves, sometimes in an unpure way of getting it from the cache collected by this crate
  • Think: release dates, toolchains and their targets (really everything rustup can manage)
• More than just stable: so far we only implemented fetching stable releases, but we care also about beta and nightly releases. We will skip the early alpha and other pre-1.0 releases.
  • stable, beta and nightly support

Considerations

• Clients (i.e. that what fetches the data) will no longer be generic
• Clients will be responsible for their own caching and incremental data fetching (it turns out that this is often quite different for different data sources)

Output / productivity

• The project will provide a set of types which describe a Rust release
• The project will provide a set of client implementations which can the data which describes a Rust release
• The project will provide a set of transformers which take the intermediate data collected by clients, and transform it into the types describing a rust release
• The project will proving one or more merge implementations which can be used to merge data collected from different data sources

[foresterre]

See also earlier work towards this: #187

[foresterre]

New code (WIP) lives at: #273

---

From that PR:

• The previous structs and traits were not great; they were defined in a sort of inverse relationship
• rust-releases-core should focus on providing structs which define the releases and their metadata.
  • I also added a "context" C generic, which allows for data to be added by specific source implementations (e.g. the rust-dist could add some s3 bucket metadata).
  • This does have a disadvantage that functions like the new merge are more complex as a result.
  • Perhaps we should leave it out initially, and add context versions of the {channel}Release structs later?
• The previous version of core focused more on fetching data from a source, while that is very much source specific. As a result, implementations almost need to work around the FetchResources trait implementation.
  • In this version, we let source impl crates deal with the fetching themselves. That is, they no longer "need" to implement a common trait to do so. Instead we focus on defining truthful structs which define a Rust release. Each impl can figure out how to build that for themselves
• We will add methods to do common operations
  • E.g. available versions so a source impl. only needs to fetch the remainder
  • E.g. push new versions (consider conflicts)
• We added support for nightly and beta releases, in addition to stable (which the current version also has)
• We focus on a consistent API surface
• We will add a way to merge two StableReleases (idem. for the other channels, and the total RustReleases) into one, so two different release sources can be combined
  • By having this context generic C, we can enhance the context on merge, e.g. merge(RustReleases<C1>, RustReleases<C2>, f) -> RustReleases<C3>
  • We do need to add a function to determine how to deal with conflicts, or when one of the two does have a value, but the other does not. -> Prototyping phase for (conflict) resolution. This is the f in the above signature.
• We will add a new release source based on Rust's github releases, which should be reasonably reliable and very fast to fetch
  • reliability: in the past it happened that releases were already published days before the merge to the stable branch was made, which results in the github release tagging
  • fetch speed: the advantage over using the S3 buckets
• This release will break the complete API

After these changes, the current roadmap is as follows:

• Release sources should give more control over (on disk) caching to the caller
• Build a static releases crate, which can act as a build in source
  • So far this was only useful if you would then be always up to date with the crate, but by allowing merges with other sources, you can merge(static, github-releases, f: prefer static) sources
• Add more examples
• Create an info cli
  • consider interoperability with rustup
• Create an info static website? (There's already one, which pulls its data from the changelog (ha: we do to): releases.rs)