Skip to content

Commit f906b4b

Browse files
committed
chore: update readme
1 parent dc44763 commit f906b4b

File tree

1 file changed

+137
-12
lines changed

1 file changed

+137
-12
lines changed

README.md

Lines changed: 137 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,53 @@
11
# Package Manager Benchmarks
22

3-
This repo contains a suite of fixutres & tools to track the performance of package managers.
3+
This repo contains a suite of fixtures & tools to track the performance of package managers. We benchmark various Node.js package managers (npm, yarn, pnpm, berry, deno, bun, vlt, nx, turbo) across different project types and scenarios.
44

5-
### Environment
5+
## Environment
66

7-
We current only test the latest `linux` runner which is the most common GitHub Action environment. The [standard GitHub-hosted public runner environment](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories) specs are below:
7+
We currently only test the latest `linux` runner which is the most common GitHub Action environment. The [standard GitHub-hosted public runner environment](https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories) specs are below:
88

99
- VM: Linux
1010
- Processor (CPU): 4
1111
- Memory (RAM): 16 GB
1212
- Storage (SSD): 14 GB
1313
- Workflow label: `ubuntu-latest`
1414

15-
We may add Mac/Windows in the future but the it will likely exponentially increase the already slow run time of the suite (**~20min**).
15+
We may add Mac/Windows in the future but it will likely exponentially increase the already slow run time of the suite (**~20min**).
1616

17-
### Configuration/Normalization
17+
## Overview
1818

19-
We do a best-effort job to configure each tool to behave as similar as possible to its peers but there's limitations to this standardization in many scenarios (as each tool makes decisions about its default support for security checks/validations/feature-set). As part of the normalization process, we count the number of packages - post-installation - & use that to determine the average speed relative to the number of packages installed. This strategy helps account for when there are significant discrepencies between the package manager's dependency graph resolution ([you can read/see more here](https://docs.google.com/presentation/d/1ojXF4jb_1MyGhew2LCbdrZ4e_0vYUr-7CoMJLJsHwZY/edit?usp=sharing)).
19+
The benchmarks measure:
20+
- Project installation times (cold and warm cache)
21+
- Task execution performance
22+
- Standard deviation of results
2023

21-
#### Example:
24+
### Project Types
25+
- Next.js
26+
- Astro
27+
- Svelte
28+
- Vue
29+
30+
### Package Managers
31+
- npm
32+
- yarn
33+
- pnpm
34+
- Yarn Berry
35+
- Deno
36+
- Bun
37+
- VLT
38+
- NX
39+
- Turbo
40+
- Node.js
41+
42+
## Configuration/Normalization
2243

44+
We do a best-effort job to configure each tool to behave as similar as possible to its peers but there's limitations to this standardization in many scenarios (as each tool makes decisions about its default support for security checks/validations/feature-set). As part of the normalization process, we count the number of packages - post-installation - & use that to determine the average speed relative to the number of packages installed. This strategy helps account for when there are significant discrepancies between the package manager's dependency graph resolution ([you can read/see more here](https://docs.google.com/presentation/d/1ojXF4jb_1MyGhew2LCbdrZ4e_0vYUr-7CoMJLJsHwZY/edit?usp=sharing)).
45+
46+
#### Example:
2347
- **Package Manager A** installs **1,000** packages in **10s** -> an avg. of **~10ms** per-package
2448
- **Package Manager B** installs **10** packages in **1s** -> an avg. of **~100ms** per-package
2549

26-
### Testing Package Installation
50+
## Testing Package Installation
2751

2852
The installation tests we run today mimic a cold-cache scenario for a variety of test fixtures (ie. we install the packages of a `next`, `vue`, `svelte` & `astro` starter project). We will likely add lockfile & warm cache tests in the near future.
2953

@@ -37,7 +61,7 @@ The installation tests we run today mimic a cold-cache scenario for a variety of
3761
- `deno`
3862
- `bun`
3963

40-
### Testing Script Execution (WIP)
64+
## Testing Script Execution (WIP)
4165

4266
This suite also tests the performance of basic script execution (ex. `npm run foo`). Notably, for any given build, test or deployment task the spawning of the process is a fraction of the overall execution time. That said, this is a commonly tracked workflow by various developer tools as it involves the common set of tasks: startup, filesystem read (`package.json`) & finally, spawning the process/command.
4367

@@ -54,6 +78,107 @@ This suite also tests the performance of basic script execution (ex. `npm run fo
5478
- `turborepo`
5579
- `nx`
5680

57-
### Output
58-
59-
Results of the test runs are found in the Actions Artifacts "results". We will eventually add a visualization of the results at some point in the future.
81+
## Running Benchmarks
82+
83+
### Local Development
84+
85+
1. Install dependencies:
86+
```bash
87+
bash ./scripts/setup.sh
88+
```
89+
90+
2. Run benchmarks:
91+
```bash
92+
# Install and benchmark a specific project
93+
bash ./scripts/install.sh <project> # cold cache
94+
bash ./scripts/install-warm.sh <project> # warm cache
95+
96+
# Run task execution benchmarks
97+
bash ./scripts/run.sh
98+
```
99+
100+
### GitHub Actions
101+
102+
The benchmarks run automatically on:
103+
- Push to main branch
104+
- Manual workflow trigger
105+
106+
The workflow:
107+
1. Sets up the environment with all package managers
108+
2. Runs benchmarks for each project type (cold and warm cache)
109+
3. Executes task performance benchmarks
110+
4. Processes results and generates visualizations
111+
5. Deploys results to GitHub Pages (main branch only)
112+
113+
## Results
114+
115+
### Console Output
116+
117+
Each benchmark run provides a summary in the console:
118+
```
119+
=== Project Name (cache type) ===
120+
package-manager: X.XXs (stddev: X.XXs)
121+
...
122+
```
123+
124+
### Generated Results
125+
126+
Results are organized by date in the `chart/results/YYYY-MM-DD/` directory:
127+
- `project-cold.json`: Cold cache installation results
128+
- `project-warm.json`: Warm cache installation results
129+
- `task-execution.json`: Task execution benchmark results
130+
- `index.html`: Interactive visualization
131+
- `styles.css`: Chart styling
132+
- `script.js`: Chart configuration
133+
134+
### Visualization
135+
136+
The generated charts show:
137+
- Installation times for each project type
138+
- Task execution performance
139+
- Standard deviation in tooltips
140+
- Missing data handling
141+
- Responsive design
142+
143+
### GitHub Pages
144+
145+
Results are automatically deployed to GitHub Pages when running on the main branch:
146+
```
147+
https://<username>.github.io/<repo>/results/
148+
```
149+
150+
Each run creates a new dated directory with its results, making it easy to track performance over time.
151+
152+
## Project Structure
153+
154+
```
155+
.
156+
├── .github/
157+
│ └── workflows/
158+
│ └── benchmark.yaml # GitHub Actions workflow
159+
├── chart/
160+
│ ├── results/ # Generated results
161+
│ ├── index.html # Chart template
162+
│ ├── styles.css # Chart styling
163+
│ └── script.js # Chart configuration
164+
├── scripts/
165+
│ ├── setup.sh # Environment setup
166+
│ ├── install.sh # Cold cache installation
167+
│ ├── install-warm.sh # Warm cache installation
168+
│ ├── run.sh # Task execution
169+
│ ├── process-results.sh # Results processing
170+
│ └── generate-chart.js # Chart generation
171+
└── results/ # Raw benchmark results
172+
```
173+
174+
## Contributing
175+
176+
1. Fork the repository
177+
2. Create your feature branch
178+
3. Commit your changes
179+
4. Push to the branch
180+
5. Create a Pull Request
181+
182+
## License
183+
184+
This project is licensed under the MIT License - see the LICENSE file for details.

0 commit comments

Comments
 (0)