How to read zip from disk directly, without loading it completely into memory?

[andronoob]

To achieve this in a NodeJS server, I implemented FileReader class as follow:

import * as zip from "@zip.js/zip.js";
import * as fsPromises from "fs/promises";

export class FileReader extends zip.Reader<undefined> {
    private readonly filePath: string;
    size = 0;

    constructor (filePath: string) {
        super(undefined);
        this.filePath = filePath;
    }

    async init(): Promise<void> {
        let stat = await fsPromises.stat(this.filePath);
        if (stat.isFile()) this.size = stat.size;
    }

    async readUint8Array(index: number, length: number): Promise<Uint8Array> {
        let result = new Uint8Array(length);
        let handle = await fsPromises.open(this.filePath, "r");
        await handle.read(result, 0, length, index);
        await handle.close();
        return result;
    }
}

I don't know whether this implementation is correct. I'm serving files inside a zip directly to HTTP/2 client (webview), however big amount of concurrent requests seemed to crash NodeJS server:

<--- Last few GCs --->

[27392:0000015622375000]    37164 ms: Mark-Compact 4038.6 (4129.2) -> 4031.1 (4138.3) MB, pooled: 0 MB, 969.75 / 0.00 ms  (average mu = 0.542, current mu = 0.282) allocation failure; scavenge might not succeed
[27392:0000015622375000]    39073 ms: Mark-Compact 4046.9 (4138.3) -> 4038.7 (4144.8) MB, pooled: 6 MB, 1863.22 / 0.00 ms  (average mu = 0.283, current mu = 0.024) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

---

Also, if I save the handle for later use (rather than calling open() and close() every time readUint8Array() is called), I cannot find a way to close it, so that NodeJS showed me the following warning:

(node:25544) Warning: Closing file descriptor 3 on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
(node:25544) [DEP0137] DeprecationWarning: Closing a FileHandle object on garbage collection is deprecated. Please close FileHandle objects explicitly using FileHandle.prototype.close(). In the future, an error will be thrown if a file descriptor is closed during garbage collection.

[gildas-lormeau]

Indeed, I can confirm that it would be great if you could close the file via a close method to be implemented in the class, for example. Can you confirm that, in this case, you don't have a memory leak problem? If so, I can add this (optional) close method in the interface.

[andronoob]

Oh sorry I didn't mean avoiding frequent close() can avoid memory issues. I guess the memory issue is probably due to big amount of concurrent requests.

[gildas-lormeau]

No problem, feel free to share the code that reads the zip file. Perhaps it would be interesting to create your Writerclass too? This could also be the cause of the problem.

EDIT: I removed part of the answer because it was off-topic.