Add README.md files to subpackages

crates/crates_io_database/README.md

@@ -0,0 +1,26 @@
+# crates_io_database
+
+This package contains the crates.io database schema as derived by `diesel print-schema`
+from the database after all the migrations have been applied.
+
+After creating new migrations (via `diesel migration generate`), you can update
+the schema by running:
+
+```sh
+diesel print-schema > crates/crates_io_database/src/schema.rs
+```
+
+## `schema.patch`
+
+Note that there is also a `schema.patch` file in this package, since the output
+of `diesel-cli` needs to be tweaked a little for our purposes. For example,
+it currently does not support printing materialized views in the same way as
+regular tables, so we have to manually add them to the schema file.
+
+If you need to update the patch file, you can do so by following these steps:
+
+1. prefix `patch_file = "src/schema.patch"` in `diesel.toml` with a `#` to comment it out.
+2. use `diesel print-schema` and save the output to `src/schema.rs.orig`
+3. use `patch -o src/schema.rs src/schema.rs.orig src/schema.patch` to apply the patch file and solve remaining issues in the `src/schema.rs` file
+4. use `diff -Naur src/schema.rs.orig src/schema.rs` to generate the new content for the `src/schema.patch` file
+5. enable the `patch_file` option in the `diesel.toml` file again.

crates/crates_io_database/src/lib.rs

@@ -1 +1,3 @@
+#![doc = include_str!("../README.md")]
+
 pub mod schema;

crates/crates_io_database_dump/README.md

@@ -0,0 +1,10 @@
+# crates_io_database_dump
+
+This package contains the code and data to create a database dump for the
+crates.io database.
+
+The most important file in this package is the `dump-db.toml` file, which
+defines how the database tables are serialized into CSV files. Specifically,
+it can be used to skip certain columns for privacy reasons, it can declare the
+serialization order of the tables, and it can declare filters, if not all rows
+should be dumped.

crates/crates_io_database_dump/src/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 use anyhow::{anyhow, Context};
 use serde::Serialize;
 use std::fs;

crates/crates_io_env_vars/README.md

@@ -0,0 +1,14 @@
+# crates_io_env_vars
+
+This package contains convenient wrappers for the `std::env::var()` function.
+
+These functions use the `dotenvy` crate to automatically load environment
+variables from a `.env` file, if it exists. This is useful for development
+environments, where you don't want to set all environment variables manually.
+
+There are also variants of the functions that make use of the `FromStr` trait to
+automatically parse the environment variables into the desired types or fail
+with corresponding error messages.
+
+Finally, there are `list()` functions that allow parsing of comma-separated
+lists of values.

crates/crates_io_env_vars/src/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 use anyhow::{anyhow, Context};
 use std::error::Error;
 use std::str::FromStr;

crates/crates_io_github/README.md

@@ -0,0 +1,12 @@
+# crates_io_github
+
+This package implements functionality for interacting with the GitHub API.
+
+It contains a `GitHubClient` trait that defines the supported operations, that
+the crates.io codebase needs to interact with GitHub. The `RealGitHubClient`
+struct is an implementation of this trait that uses the `reqwest` crate to
+perform the actual HTTP requests.
+
+If the `mock` feature is enabled, a `MockGitHubClient` struct is available,
+which can be used for testing purposes. This struct is generated automatically
+by the [`mockall`](https://docs.rs/mockall) crate.

crates/crates_io_github/src/lib.rs

@@ -1,4 +1,4 @@
-//! This module implements functionality for interacting with GitHub.
+#![doc = include_str!("../README.md")]
 
 #[macro_use]
 extern crate tracing;

crates/crates_io_index/README.md

@@ -0,0 +1,11 @@
+# crates_io_index
+
+This package contains the code necessary to interact with the
+[crates.io-index repository](https://github.com/rust-lang/crates.io-index).
+
+Specifically, it contains:
+
+- the data structures used to serialize and deserialize the files in the index
+- a `Repository` abstraction to perform various operations on the index
+- and, for testing purposes, an `UpstreamIndex` struct that can be used to
+  create a fake index locally.

crates/crates_io_index/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("README.md")]
+
 #[macro_use]
 extern crate serde;
 #[macro_use]

crates/crates_io_markdown/README.md

@@ -0,0 +1,5 @@
+# crates_io_markdown
+
+This package contains the code to render markdown files into raw HTML for the
+crates.io website. This functionality is used to render the `README.md` files of
+crates, so that they can be displayed on the crate's page.

crates/crates_io_markdown/lib.rs

@@ -1,4 +1,4 @@
-//! Render Markdown files to HTML.
+#![doc = include_str!("README.md")]
 
 use ammonia::{Builder, UrlRelative, UrlRelativeEvaluate};
 use comrak::nodes::{AstNode, NodeValue};

crates/crates_io_pagerduty/README.md

@@ -0,0 +1,7 @@
+# crates_io_pagerduty
+
+This package contains the code necessary to interact with the PagerDuty API.
+
+The crates.io on-call team uses PagerDuty to get notified about incidents. This
+package contains a `PagerdutyClient` struct that can be configured with an API
+token and service key to then trigger, acknowledge, and resolve incidents.

crates/crates_io_pagerduty/src/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 use anyhow::{anyhow, Result};
 use reqwest::{header, Client, StatusCode as Status};
 use secrecy::{ExposeSecret, SecretString};

crates/crates_io_session/README.md

@@ -0,0 +1,7 @@
+# crates_io_session
+
+This package contains a `SessionExtension` extractor for the
+[`axum`](https://docs.rs/axum) web framework and a corresponding
+`attach_session()` middleware based on a signed `cargo_session` cookie.
+This abstraction allows us to save and retrieve data from the session
+cookie in a safe way.

crates/crates_io_session/src/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 use axum::extract::{Extension, FromRequestParts, Request};
 use axum::middleware::Next;
 use axum::response::{IntoResponse, Response};

crates/crates_io_smoke_test/README.md

@@ -0,0 +1,11 @@
+# crates_io_smoke_test
+
+This package contains a basic smoke test for the <https://staging.crates.io>
+environment. It uses the API to fetch the metadata of a test crate, publishes
+a new patch version, attempts to download the crate file, and checks that the
+git and sparse indexes contain the new version.
+
+Note that a valid `CARGO_REGISTRY_TOKEN` environment variable is required to
+run the smoke test. This token must have the `publish` permission for the test
+crate. If `--skip-publish` is passed, the smoke test will not publish a new
+version of the test crate.

crates/crates_io_smoke_test/src/main.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 mod api;
 mod cargo;
 mod exit_status_ext;

crates/crates_io_tarball/README.md

@@ -0,0 +1,13 @@
+# crates_io_tarball
+
+This package is used to extract metadata from a `.crate` file, which is the
+format used to distribute Rust libraries on https://crates.io.
+
+The main source of metadata is the `Cargo.toml` file, which must be included in
+the `.crate` file.
+
+A secondary source of metadata is the `.cargo_vcs_info.json` file, which
+contains information about the version control system that was used at the
+time of publishing the crate. Note that this file is optional, and must not be
+relied upon for critical information since a malicious user could tamper with
+it before publishing the crate.

crates/crates_io_tarball/src/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 #[cfg(test)]
 #[macro_use]
 extern crate claims;

crates/crates_io_team_repo/README.md

@@ -0,0 +1,12 @@
+# crates_io_team_repo
+
+The code in this package interacts with the
+<https://github.com/rust-lang/team/> repository.
+
+The `TeamRepo` trait is used to abstract away the HTTP client for testing
+purposes. The `TeamRepoImpl` struct is the actual implementation of
+the trait.
+
+If the `mock` feature is enabled, a `MockTeamRepo` struct is available,
+which can be used for testing purposes. This struct is generated automatically
+by the [`mockall`](https://docs.rs/mockall) crate.

crates/crates_io_team_repo/src/lib.rs

@@ -1,9 +1,4 @@
-//! The code in this module interacts with the
-//! <https://github.com/rust-lang/team/> repository.
-//!
-//! The [TeamRepo] trait is used to abstract away the HTTP client for testing
-//! purposes. The [TeamRepoImpl] struct is the actual implementation of
-//! the trait.
+#![doc = include_str!("../README.md")]
 
 use async_trait::async_trait;
 use reqwest::{Certificate, Client};

crates/crates_io_test_db/README.md

@@ -0,0 +1,15 @@
+# crates_io_test_db
+
+This package contains the code necessary to create test databases for testing
+purposes.
+
+`TestDatabase::new()` can be used to create a new test database, based on a
+template database, which is lazily created the first time it is needed.
+
+The databases are created based on the `TEST_DATABASE_URL` environment variable,
+which should be set to a valid database URL. The template database will then be
+created with a similar name and `_template` suffix, while the test databases
+will use random suffixes.
+
+Note that the template database will be created with applied database migrations,
+so if you need an empty database, this is not the right tool for you.

crates/crates_io_test_db/src/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 use crates_io_env_vars::required_var_parsed;
 use diesel::prelude::*;
 use diesel::r2d2::{ConnectionManager, Pool, PooledConnection};

crates/crates_io_worker/README.md

@@ -0,0 +1,12 @@
+# crates_io_worker
+
+This package contains the background job runner for the crates.io application.
+
+The implementation was originally extracted from crates.io into the separate
+[`swirl`](https://github.com/sgrif/swirl) project, but has since been
+re-integrated and heavily modified.
+
+The background worker uses a `background_jobs` PostgreSQL table to store jobs
+that need to be run. Once a job is picked up by a worker, the table row is
+locked, and the job is run. If the job fails, it will be retried with
+exponential backoff. If the job succeeds, the row will be deleted.

crates/crates_io_worker/src/lib.rs

@@ -1,3 +1,5 @@
+#![doc = include_str!("../README.md")]
+
 mod background_job;
 mod errors;
 mod job_registry;