Skip to content

Commit 674aa9d

Browse files
bmiddhabmiddha
authored
[zipsync] Add new tool to efficiently pack and unpack cache entries (#5361)
1 parent a44b38c commit 674aa9d

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+3194
-17
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ These GitHub repositories provide supplementary resources for Rush Stack:
5757
| [/apps/rush](./apps/rush/) | [![npm version](https://badge.fury.io/js/%40microsoft%2Frush.svg)](https://badge.fury.io/js/%40microsoft%2Frush) | [changelog](./apps/rush/CHANGELOG.md) | [@microsoft/rush](https://www.npmjs.com/package/@microsoft/rush) |
5858
| [/apps/rush-mcp-server](./apps/rush-mcp-server/) | [![npm version](https://badge.fury.io/js/%40rushstack%2Fmcp-server.svg)](https://badge.fury.io/js/%40rushstack%2Fmcp-server) | [changelog](./apps/rush-mcp-server/CHANGELOG.md) | [@rushstack/mcp-server](https://www.npmjs.com/package/@rushstack/mcp-server) |
5959
| [/apps/trace-import](./apps/trace-import/) | [![npm version](https://badge.fury.io/js/%40rushstack%2Ftrace-import.svg)](https://badge.fury.io/js/%40rushstack%2Ftrace-import) | [changelog](./apps/trace-import/CHANGELOG.md) | [@rushstack/trace-import](https://www.npmjs.com/package/@rushstack/trace-import) |
60+
| [/apps/zipsync](./apps/zipsync/) | [![npm version](https://badge.fury.io/js/%40rushstack%2Fzipsync.svg)](https://badge.fury.io/js/%40rushstack%2Fzipsync) | [changelog](./apps/zipsync/CHANGELOG.md) | [@rushstack/zipsync](https://www.npmjs.com/package/@rushstack/zipsync) |
6061
| [/eslint/eslint-bulk](./eslint/eslint-bulk/) | [![npm version](https://badge.fury.io/js/%40rushstack%2Feslint-bulk.svg)](https://badge.fury.io/js/%40rushstack%2Feslint-bulk) | [changelog](./eslint/eslint-bulk/CHANGELOG.md) | [@rushstack/eslint-bulk](https://www.npmjs.com/package/@rushstack/eslint-bulk) |
6162
| [/eslint/eslint-config](./eslint/eslint-config/) | [![npm version](https://badge.fury.io/js/%40rushstack%2Feslint-config.svg)](https://badge.fury.io/js/%40rushstack%2Feslint-config) | [changelog](./eslint/eslint-config/CHANGELOG.md) | [@rushstack/eslint-config](https://www.npmjs.com/package/@rushstack/eslint-config) |
6263
| [/eslint/eslint-patch](./eslint/eslint-patch/) | [![npm version](https://badge.fury.io/js/%40rushstack%2Feslint-patch.svg)](https://badge.fury.io/js/%40rushstack%2Feslint-patch) | [changelog](./eslint/eslint-patch/CHANGELOG.md) | [@rushstack/eslint-patch](https://www.npmjs.com/package/@rushstack/eslint-patch) |

‎apps/zipsync/.npmignore‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# THIS IS A STANDARD TEMPLATE FOR .npmignore FILES IN THIS REPO.
2+
3+
# Ignore all files by default, to avoid accidentally publishing unintended files.
4+
*
5+
6+
# Use negative patterns to bring back the specific things we want to publish.
7+
!/bin/**
8+
!/lib/**
9+
!/lib-*/**
10+
!/dist/**
11+
12+
!CHANGELOG.md
13+
!CHANGELOG.json
14+
!heft-plugin.json
15+
!rush-plugin-manifest.json
16+
!ThirdPartyNotice.txt
17+
18+
# Ignore certain patterns that should not get published.
19+
/dist/*.stats.*
20+
/lib/**/test/
21+
/lib-*/**/test/
22+
*.test.js
23+
24+
# NOTE: These don't need to be specified, because NPM includes them automatically.
25+
#
26+
# package.json
27+
# README.md
28+
# LICENSE
29+
30+
# ---------------------------------------------------------------------------
31+
# DO NOT MODIFY ABOVE THIS LINE! Add any project-specific overrides below.
32+
# ---------------------------------------------------------------------------

‎apps/zipsync/LICENSE‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
@rushstack/zipsync
2+
3+
Copyright (c) Microsoft Corporation. All rights reserved.
4+
5+
MIT License
6+
7+
Permission is hereby granted, free of charge, to any person obtaining
8+
a copy of this software and associated documentation files (the
9+
"Software"), to deal in the Software without restriction, including
10+
without limitation the rights to use, copy, modify, merge, publish,
11+
distribute, sublicense, and/or sell copies of the Software, and to
12+
permit persons to whom the Software is furnished to do so, subject to
13+
the following conditions:
14+
15+
The above copyright notice and this permission notice shall be
16+
included in all copies or substantial portions of the Software.
17+
18+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
19+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
20+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
21+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
22+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
23+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
24+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

‎apps/zipsync/README.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# @rushstack/zipsync
2+
3+
zipsync is a focused tool for packing and unpacking build cache entries using a constrained subset of the ZIP format for high performance. It optimizes the common scenario where most files already exist in the target location and are unchanged.
4+
5+
## Goals & Rationale
6+
7+
- **Optimize partial unpack**: Most builds reuse the majority of previously produced outputs. Skipping rewrites preserves filesystem and page cache state.
8+
- **Only write when needed**: Fewer syscalls.
9+
- **Integrated cleanup**: Removes the need for a separate `rm -rf` pass; extra files and empty directories are pruned automatically.
10+
- **ZIP subset**: Compatibility with malware scanners.
11+
- **Fast inspection**: The central directory can be enumerated without inflating the entire archive (unlike tar+gzip).
12+
13+
## How It Works
14+
15+
### Pack Flow
16+
17+
```
18+
for each file F
19+
write LocalFileHeader(F)
20+
stream chunks:
21+
read -> hash + crc + maybe compress -> write
22+
finalize compressor
23+
write DataDescriptor(F)
24+
add metadata entry (same pattern)
25+
write central directory records
26+
```
27+
28+
### Unpack Flow
29+
30+
```
31+
load archive -> parse central dir -> read metadata
32+
scan filesystem & delete extraneous entries
33+
for each entry (except metadata):
34+
if unchanged (sha1 matches) => skip
35+
else extract (decompress if needed)
36+
```
37+
38+
## Why ZIP (vs tar + gzip)
39+
40+
Pros for this scenario:
41+
42+
- Central directory enables cheap listing without decompressing entire payload.
43+
- Widely understood / tooling-friendly (system explorers, scanners, CI tooling).
44+
- Per-file compression keeps selective unpack simple (no need to inflate all bytes).
45+
46+
Trade-offs:
47+
48+
- Tar+gzip can exploit cross-file redundancy for better compressed size in datasets with many similar files.

‎apps/zipsync/bin/zipsync‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#!/usr/bin/env node
2+
require('../lib/start.js');

‎apps/zipsync/config/jest.config.json‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
{
2+
"extends": "local-node-rig/profiles/default/config/jest.config.json",
3+
"setupFilesAfterEnv": ["<rootDir>/config/jestSymbolDispose.js"]
4+
}

‎apps/zipsync/config/jestSymbolDispose.js‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
2+
// See LICENSE in the project root for license information.
3+
4+
const disposeSymbol = Symbol('Symbol.dispose');
5+
const asyncDisposeSymbol = Symbol('Symbol.asyncDispose');
6+
7+
Symbol.asyncDispose ??= asyncDisposeSymbol;
8+
Symbol.dispose ??= disposeSymbol;

‎apps/zipsync/config/rig.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
// The "rig.json" file directs tools to look for their config files in an external package.
3+
// Documentation for this system: https://www.npmjs.com/package/@rushstack/rig-package
4+
"$schema": "https://developer.microsoft.com/json-schemas/rig-package/rig.schema.json",
5+
6+
"rigPackageName": "local-node-rig"
7+
}

‎apps/zipsync/eslint.config.js‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
// Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT license.
2+
// See LICENSE in the project root for license information.
3+
4+
const nodeTrustedToolProfile = require('local-node-rig/profiles/default/includes/eslint/flat/profile/node-trusted-tool');
5+
const friendlyLocalsMixin = require('local-node-rig/profiles/default/includes/eslint/flat/mixins/friendly-locals');
6+
7+
module.exports = [
8+
...nodeTrustedToolProfile,
9+
...friendlyLocalsMixin,
10+
{
11+
files: ['**/*.ts', '**/*.tsx'],
12+
languageOptions: {
13+
parserOptions: {
14+
tsconfigRootDir: __dirname
15+
}
16+
}
17+
}
18+
];

‎apps/zipsync/package.json‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
{
2+
"name": "@rushstack/zipsync",
3+
"version": "0.0.0",
4+
"description": "CLI tool for creating and extracting ZIP archives with intelligent filesystem synchronization",
5+
"repository": {
6+
"type": "git",
7+
"url": "https://github.com/microsoft/rushstack.git",
8+
"directory": "apps/zipsync"
9+
},
10+
"bin": {
11+
"zipsync": "./bin/zipsync"
12+
},
13+
"license": "MIT",
14+
"scripts": {
15+
"start": "node lib/start",
16+
"build": "heft build --clean",
17+
"_phase:build": "heft run --only build -- --clean",
18+
"_phase:test": "heft run --only test -- --clean"
19+
},
20+
"dependencies": {
21+
"@rushstack/terminal": "workspace:*",
22+
"@rushstack/ts-command-line": "workspace:*",
23+
"typescript": "~5.8.2",
24+
"@rushstack/lookup-by-path": "workspace:*"
25+
},
26+
"devDependencies": {
27+
"@rushstack/heft": "workspace:*",
28+
"eslint": "~9.25.1",
29+
"local-node-rig": "workspace:*"
30+
}
31+
}

0 commit comments

Comments
 (0)