-
Notifications
You must be signed in to change notification settings - Fork 751
deps(ziputil): migrate ziputil to use a web-safe and typed ZipStream #6816
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
/runIntegrationTests |
| } | ||
|
|
||
| public static async unzip(zipBuffer: Buffer): Promise<Entry[]> { | ||
| const reader = new ZipReader(new Uint8ArrayReader(new Uint8Array(zipBuffer))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
off topic: this doesn't actually "stream" the result, it all goes into memory, right? that's something we need to address eventually.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I noticed this as well. Also, the current logic in zipUtils dumps this to the file system before uploading whereas prepareRepoData uploads the buffer itself.
If we can centralize the uploading logic (also has two completely separate implementations), this streaming change will be much easier.
|
#6566 is apparently still an issue since the fix only lives in |
8b3dcb6 to
6d78da8
Compare
6d78da8 to
f88c80b
Compare
Problem
This PR solves two very related problems so there are done together in one PR:
ZipUtil uses
admZipwhich not web compatible. Also, the larger issue is thatZipUtilduplicatesprepareRepoDatain some ways, and it is difficult to align their implementations if they use different libraries.The ZipStream utility is a wrapper around the
zip.jslibrary. However, it imports it viaCommonJSrequiresyntax causing us to lose type information. This is dangerous because we are assuming properties and functions exist within the library, and are blind to changes in the underlying library. This makes theZipStreammodule difficult and unsafe to work with.Solution
ZipUtilaway fromadmZip.ZipUtiltests to live incorepackage sinceZipUtillives there.ZipStream+ tests for them.zip.jsas a module and tell the compiler to ignore it. It looks like this happens becausezip.jsdoesn't support CommonJS, and in general doesn't support Node officially (source: require() of ES modules is not supported. gildas-lormeau/zip.js#362). However, a future commit appears to add this support (on that same issue) so unsure why we still get the error. Experimentally this is working, so its likely Webpack handles this for us.also addresses: #6566
Verification
feature/xbranches will not be squash-merged at release time.