From 5ef881c7c7eda488a0b6a58d011217959d805912 Mon Sep 17 00:00:00 2001 From: Tobias Bieniek Date: Sun, 27 Jul 2025 18:45:52 +0200 Subject: [PATCH 1/4] docs: Remove outdated version requirements --- docs/CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 0afa7ca1ad9..9a6e52c86cf 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -104,7 +104,7 @@ as well. In order to run the frontend on Windows and macOS, you will need to have installed: -- [node](https://nodejs.org/en/) >= 18.16.0 (see `package.json` and `.github/workflows/ci.yml` for what we currently use) +- [node](https://nodejs.org/en/) (see `package.json` engines field for the exact version required) - [pnpm](https://pnpm.io) >= 8.5.1 Follow the links for each of these tools for their recommended installation @@ -155,7 +155,7 @@ pnpm test In order to run the backend, you will need to have installed: -- [Rust](https://www.rust-lang.org/en-US/) stable >= 1.56.0 and cargo, which comes with Rust +- [Rust](https://www.rust-lang.org/en-US/) stable (see `rust-toolchain.toml` for version) and cargo, which comes with Rust - [Postgres](https://www.postgresql.org/) >= 9.5 - [OpenSSL](https://www.openssl.org/) >= 1.0.2k - [diesel_cli](http://diesel.rs/guides/getting-started/) >= 2.0.0 and < 3.0.0 From 257b8070b9ac4a4c33ac7502935f5bff8644f2e0 Mon Sep 17 00:00:00 2001 From: Tobias Bieniek Date: Sun, 27 Jul 2025 18:46:23 +0200 Subject: [PATCH 2/4] docs: Replace outdated JSLint mention with ESLint --- docs/CONTRIBUTING.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 9a6e52c86cf..f24469d0c8e 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -36,7 +36,7 @@ a documentation comment on it, it'd be great if you could add one to it too. When you submit a pull request, it will be automatically tested on GitHub Actions. In addition to running both the frontend and the backend tests described below, -GitHub Actions runs [jslint], [clippy], and [rustfmt] on each PR. +GitHub Actions runs [ESLint], [clippy], and [rustfmt] on each PR. If you don't want to run these tools locally, please watch the GitHub Actions results and submit additional commits to your pull request to fix any issues they find! @@ -47,7 +47,7 @@ instructions and the [.github/workflows/ci.yml] file in this repository for the installation and running instructions. The logs for recent builds in GitHub Actions may also be helpful to see which versions of these tools we're currently using. -[jslint]: http://jslint.com/ +[ESLint]: https://eslint.org/ [clippy]: https://github.com/rust-lang-nursery/rust-clippy [rustfmt]: https://github.com/rust-lang-nursery/rustfmt [.github/workflows/ci.yml]: /.github/workflows/ci.yml From ea50b3974af799d882db01594b2601f07f5c81ec Mon Sep 17 00:00:00 2001 From: Tobias Bieniek Date: Sun, 27 Jul 2025 18:47:03 +0200 Subject: [PATCH 3/4] docs: Remove empty frontend document --- docs/ARCHITECTURE.md | 3 +-- docs/FRONTEND.md | 7 ------- 2 files changed, 1 insertion(+), 9 deletions(-) delete mode 100644 docs/FRONTEND.md diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index f9b41442888..4fa480e5349 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -37,8 +37,7 @@ The backend stores information in a Postgres database. ## Frontend - Ember.js -The frontend of crates.io is written in JavaScript using [Ember.js][]. More information about the -frontend is in [`docs/FRONTEND.md`](https://github.com/rust-lang/crates.io/blob/master/docs/FRONTEND.md). +The frontend of crates.io is written in JavaScript using [Ember.js][]. More information about frontend development is in [`docs/CONTRIBUTING.md`](https://github.com/rust-lang/crates.io/blob/master/docs/CONTRIBUTING.md). [Ember.js]: https://emberjs.com/ diff --git a/docs/FRONTEND.md b/docs/FRONTEND.md deleted file mode 100644 index 38c745c3781..00000000000 --- a/docs/FRONTEND.md +++ /dev/null @@ -1,7 +0,0 @@ -# Frontend Overview - -The frontend of crates.io is written in JavaScript using [Ember.js]. Most of that code lives in -the `app` directory. We endeavor to follow Ember conventions and best practices, but we're Rust -developers, so we don't always live up to this goal :) - -[Ember.js]: https://emberjs.com/ From ab74432b820a6443d993fd05365ba847578088f8 Mon Sep 17 00:00:00 2001 From: Tobias Bieniek Date: Sun, 27 Jul 2025 18:48:04 +0200 Subject: [PATCH 4/4] docs: Remove obsolete/redundant backend document --- docs/ARCHITECTURE.md | 4 +- docs/BACKEND.md | 199 ------------------------------------------- 2 files changed, 2 insertions(+), 201 deletions(-) delete mode 100644 docs/BACKEND.md diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 4fa480e5349..3b22cc44658 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -18,8 +18,8 @@ Documentation about the codebase appears in these locations: The backend of crates.io is written in Rust. Most of that code lives in the _src_ directory. It serves a JSON API over HTTP, and the HTTP server interface is provided by the [axum][] crate and -related crates. More information about the backend is in -[`docs/BACKEND.md`](https://github.com/rust-lang/crates.io/blob/master/docs/BACKEND.md). +related crates. More information about backend development is in +[`docs/CONTRIBUTING.md`](https://github.com/rust-lang/crates.io/blob/master/docs/CONTRIBUTING.md). [axum]: https://crates.io/crates/axum diff --git a/docs/BACKEND.md b/docs/BACKEND.md deleted file mode 100644 index 050fea9d3f2..00000000000 --- a/docs/BACKEND.md +++ /dev/null @@ -1,199 +0,0 @@ -# Backend Overview - -## Server - -The code to actually run the server is in _src/bin/server.rs_. This is where most of the pieces of -the system are instantiated and configured, and can be thought of as the "entry point" to crates.io. - -The server does the following things: - -1. Initialize logging -2. Check out the index git repository, if it isn't already checked out -3. Reads values from environment variables to configure a new instance of `crates_io::App` -4. Adds middleware to the app by calling `crates_io::middleware` -5. Syncs the categories defined in _src/categories.toml_ with the categories in the database -6. Starts a [hyper] server that uses the `crates_io::App` instance -7. Tells Nginx on Heroku that the application is ready to receive requests, if running on Heroku -8. Blocks forever (or until the process is killed) - -[hyper]: https://crates.io/crates/hyper - -## Routes - -The API URLs that the server responds to (aka "routes") are defined in -_src/router.rs_. - -All of the `api_router` routes are mounted under the `/api/v1` path (see the -lines that look like `router.get("/api/v1/*path", R(api_router.clone()));`). - -Each API route definition looks like this: - -```rust -api_router.get("/crates", C(krate::index)); -``` - -This line defines a route that responds to a GET request made to -`/api/v1/crates` with the results of calling the `krate::index` function. `C` -is a struct that holds a function and implements the [`conduit::Handler`][] -trait so that the results of the function are the response if the function -succeeds, and that the server returns an error response if the function doesn't -succeed. The `C` struct's purpose is to reduce some boilerplate. - -[`conduit::Handler`]: https://docs.rs/conduit/0.8.1/conduit/trait.Handler.html - -## Code having to do with running a web application - -These modules could _maybe_ be refactored into another crate. Maybe not. But their primary purpose -is supporting the running of crates.io's web application parts, and they don't have much to do with -the crate registry purpose of the application. - -### The `app` module - -This contains the `App` struct, which holds a `Config` instance plus a few more application -components such as: - -- The database connection pools (there are two until we finish migrating the app to use Diesel - everywhere) -- The GitHub OAuth configuration -- The cookie session key given to [conduit-cookie][] -- The `git2::Repository` instance for the index repo checkout -- The `Config` instance - -This module also contains `AppMiddleware`, which implements the `Middleware` trait in order to -inject the `app` instance into every request. That way, we can call `req.app()` to get to any of -these components. - -[conduit-cookie]: https://crates.io/crates/conduit-cookie - -### The `config` module - -### The `db` module - -### The `dist` module - -### The `http` module - -### The `model` module - -### The `schema` module - -### The `utils` module - -## Code having to do with managing a registry of crates - -These modules are specific to the domain of being a crate registry. These concepts would exist no -matter what language or framework crates.io was implemented in. - -### The `krate` module - -### The `users` module - -### The `badge` module - -### The `categories` module - -### The `category` module - -### The `dependency` module - -### The `download` module - -### The `git` module - -### The `keyword` module - -### The `owner` module - -### The `upload` module - -### The `uploaders` module - -### The `version` module - -## Database - -## Tests - -### Integration tests - -A suite of integration tests that require a full crates.io app to be -instantiated and available live in `src/tests`. These are a mixture of tests -that exercise routes and controllers like normal API consumers, and other tests -that require a full blown application and database to be available. - -Tests that interact with HTTP services — for example, S3 to upload packages or -index entries — do so by routing requests through a HTTP proxy configured within -`TestApp`. This proxy implements similar behaviour to the [`vcr` Ruby -gem](https://github.com/vcr/vcr): by default, requests are not forwarded to the -actual upstream service, but instead the request is checked against a -"cassette", which is a JSON file in `src/tests/http-data` that contains the -expected request and response. If the request matches, then the saved response -is returned. If the request doesn't match, then an error is returned and the -test fails. - -#### Updating test cassettes - -When updating integration tests that make HTTP requests, you may need to update -the test cassette associated with that test. This is controlled by the `RECORD` -environment variable: when set to `yes`, the request will actually be forwarded -to the upstream service, and the cassette will be updated with the new response. - -In addition to setting `RECORD`, you will also need to set a number of other -environment variables to handle any requests to S3. These are in `.env.sample`, -but are also reproduced here for convenience: - -- `TEST_S3_BUCKET`: the S3 bucket used for package uploads. -- `TEST_S3_REGION`: the S3 region used to package uploads. This may also be an - absolute URL (such as `http://127.0.0.1:19000`), in which that will be used as - the endpoint. If this is an S3 region name, then it must be one that supports - what [AWS documents as `s3-Region` host names][s3-region]. -- `TEST_S3_INDEX_BUCKET`: the S3 bucket used for sparse index uploads. -- `TEST_S3_INDEX_REGION`: the S3 region used for sparse index uploads. This has - the same semantics as `TEST_S3_REGION`. -- `TEST_AWS_ACCESS_KEY`: the AWS access key used with S3. -- `TEST_AWS_SECRET_KEY`: the AWS secret key used with S3. - -Note that, if the bucket and region environment variables are set, they _must_ -match the values used when `src/tests/http-data` was regenerated for existing -tests to pass. - -In practice, we use [Minio](https://min.io/) to mock the S3 API when generating -test cassettes, since this means we don't have to share S3 credentials or pay -for an actual bucket. You can run Minio locally on port 19000 with `crates-test` -and `crates-index-test` buckets to match the existing test cassettes. To do this -in Docker with the access key `minio` and secret key `miniominio`, run the -following command: - -```sh -docker run --rm \ - -p 19000:9000 -p 19001:9001 \ - -e MINIO_ROOT_USER=minio -e MINIO_ROOT_PASSWORD=miniominio \ - --entrypoint /bin/sh \ - quay.io/minio/minio \ - -c 'mkdir /data/crates-test /data/crates-index-test && exec env minio server /data --console-address :9001' -``` - -This will give you an S3 compatible Minio instance listening on port 19000, and -a management console on port 19001. - -#### Running tests against S3 - -If you want to run the test suite against an actual S3 bucket, you must do so -with the `RECORD` environment variable set to `yes`, otherwise tests will fail -before any requests are actually sent because the URL in the test cassette -doesn't match the request URL. For example: - -```sh -RECORD=yes \ - TEST_S3_REGION=us-east-1 TEST_S3_BUCKET=my-test-bucket \ - TEST_S3_INDEX_REGION=us-east-1 TEST_S3_INDEX_BUCKET=my-test-bucket \ - TEST_AWS_ACCESS_KEY=an-actual-AWS-key TEST_AWS_SECRET_KEY=an-actual-secret-key \ - cargo test -``` - -Please do not commit any updated test cassettes generated as a result of running -the test suite against S3. - -## Scripts - -[s3-region]: https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#s3-dash-region