Question about ReadableStream<Uint8Array>

[Ercilan]

Using 2.8.3.
I am pulling files from an Android device via https://github.com/yume-chan/ya-webadb
and compressing them at the same time. Some key code:

// return ReadableStream<Uint8Array>
const stream = adb.sync().read(file.path);

const transform = new TransformStream<Uint8Array, Uint8Array>({
  transform(chunk, controller) {
    console.log(file.relativePath, chunk);
    controller.enqueue(chunk);
  },
});
const transformedStream = stream.pipeThrough(transform);
await zipWriter.add(file.relativePath, transformedStream);

I added a TransformStream to print debug logs and found that the Uint8Array data view for each file contains the correct range of data. And their underlying buffers actually include the adb sync protocol headers and footers. For example, this 'Hello World' text file:

This seems to cause zip.js to compress the wrong data when using the buffer during compression.

There are some strange points:

First, I need to compress multiple zip files along with some text files. After zip.js compresses multiple files from adb sync, I tested extracting them and found that only the data from the txt files is abnormal after decompression.

For the extracted txt files, only the leading 'DATAxxx' is included; the end is correctly truncated without 'DONE'. Also, this error occurs intermittently—sometimes it works fine, and other times the files have incorrect data at the beginning.

It’s possible that all the data being compressed contains adb protocol headers and footers, but during extraction some files appear normal while others are corrupted? I don’t really understand how zip compression works in this regard.

Is this a bug? So, aside from manually copying a Uint8Array that doesn’t share the underlying buffer (which seems to work), is there a better solution? Thanks.

If you don’t have any ideas or thoughts, I can take the time to try creating a reproduction.

[gildas-lormeau]

I need to do some investigation I think. Out of curiosity, does it work ok if you replace controller.enqueue(chunk); with controller.enqueue(new Uint8Array(chunk));? Also could you tell me if importing /index-native.js from zip.js (instead of the default index file) has any effect?

[gildas-lormeau]

It would be also interesting to know if setting the option useWebWorkersto false as any impact, the same for the option useCompressionStream, and the option bufferedWrite (set to true).

[Ercilan]

I've reproduced this issue, and it appears to be confirmed as a problem with zip.js. You can use this file for any debugging tests. https://gist.github.com/Ercilan/6965293c1ee26eabc4e31bda41b6783d

[Ercilan]

new Uint8Array, like chunk.slice(), does not share the underlying buffer, so the result should be normal.
But I still feel this isn't the best approach. Perhaps zip.js should directly retrieve data from the correct index range of the buffer during compression to avoid copying?

[gildas-lormeau]

Your test is a bit complicated. Is it possible to isolate more the issue or simplify the test code? For example, what should I change in the code below in order to reproduce it?

const TEXT_CONTENT = "Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Typi non habent claritatem insitam; est usus legentis in iis qui facit eorum claritatem. Investigationes demonstraverunt lectores legere me lius quod ii legunt saepius. Claritas est etiam processus dynamicus, qui sequitur mutationem consuetudium lectorum. Mirum est notare quam littera gothica, quam nunc putamus parum claram, anteposuerit litterarum formas humanitatis per seacula quarta decima et quinta decima. Eodem modo typi, qui nunc nobis videntur parum clari, fiant sollemnes in futurum.";
const INPUT_DATA = new TextEncoder().encode(TEXT_CONTENT);
const FILENAME = "lorem.txt";

await test();

async function test() {
	const chunkedStream = new TransformStream();
	const writer = chunkedStream.writable.getWriter();
	const zipStream = new TransformStream();
	const zipWriter = new zip.ZipWriter(zipStream);
	const zipReader = new zip.ZipReader(zipStream);
	const [entries] = await Promise.all([
		zipReader.getEntries(),
		zipWriter.add(FILENAME, chunkedStream),
		(async () => {
			const chunkSize = 30;
			for (let offset = 0; offset < INPUT_DATA.length; offset += chunkSize) {
				const chunk = INPUT_DATA.subarray(offset, offset + chunkSize);
				await writer.ready;
				await writer.write(chunk);
			}
			await writer.close();
			await zipWriter.close();
		})()
	]);
	const firstEntry = entries.shift();
	const entryStream = new TransformStream();
	const [entryText] = await Promise.all([
		new Response(entryStream.readable).text(),
		firstEntry.getData(entryStream)
	]);
	await zipReader.close();
	await zip.terminateWorkers();
	if (TEXT_CONTENT != entryText) {
		throw new Error("Invalid entry data");
	}
}

[Ercilan]

That HTML replicates my actual logic, the asynchronous generator produces something akin to a complete file data structure: new Uint8Array(buffer, byteOffset, length);. Each file is then converted into a stream and added sequentially via zipWriter.add.

const TEXT_CONTENT =
    "Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Typi non habent claritatem insitam; est usus legentis in iis qui facit eorum claritatem. Investigationes demonstraverunt lectores legere me lius quod ii legunt saepius. Claritas est etiam processus dynamicus, qui sequitur mutationem consuetudium lectorum. Mirum est notare quam littera gothica, quam nunc putamus parum claram, anteposuerit litterarum formas humanitatis per seacula quarta decima et quinta decima. Eodem modo typi, qui nunc nobis videntur parum clari, fiant sollemnes in futurum.";
const DATA_IN_BUFFER = `junk_${TEXT_CONTENT}_junk`;

const INPUT_DATA = new TextEncoder().encode(DATA_IN_BUFFER);
const FILENAME = "lorem.txt";

await test();

async function test() {
  const chunkedStream = new TransformStream();
  const writer = chunkedStream.writable.getWriter();
  const zipStream = new TransformStream();
  const zipWriter = new zip.ZipWriter(zipStream);
  const zipReader = new zip.ZipReader(zipStream);

  const [entries] = await Promise.all([
    zipReader.getEntries(),
    zipWriter.add(FILENAME, chunkedStream),
    (async () => {
      await writer.ready;
      const data = INPUT_DATA.subarray(5, INPUT_DATA.length - 5);
      await writer.write(data);
      await writer.close();
      await zipWriter.close();
    })(),
  ]);

  const firstEntry = entries.shift();
  const entryStream = new TransformStream();
  const [entryText] = await Promise.all([
    new Response(entryStream.readable).text(),
    firstEntry.getData(entryStream),
  ]);

  await zipReader.close();
  await zip.terminateWorkers();

  console.log("TEXT_CONTENT:", TEXT_CONTENT);
  console.log("Entry text:", entryText);
  if (TEXT_CONTENT != entryText) {
    throw new Error("Invalid entry data");
  }

  return entryText;
}

[gildas-lormeau]

Thank you, I was able to reproduce and fix the issue. The fix is available in the version 2.8.6 I've just released.

[Ercilan]

👍Alright, it seems I was close to uncovering the truth. During initial testing, I only traced it to the getWebWorker call above. Since it occurred sporadically in my actual project, I assumed it was happening within the web worker and didn't investigate further.
One point for performance optimization (nitpicky): Is there a way to add an option to avoid this copying? For example, in my case, the buffer is no longer important—it could be moved entirely to the web worker, completely bypassing the copy.

[gildas-lormeau]

You were right to point that out. I just realized that the ArrayBuffer#slice call could be avoided by passing to the web worker the Uint8Array instance rather than its ArrayBuffer buffer, which is passed as transferable object. As a result, with the version 2.8.7 I've just released, there is no longer any data copying when data is passed to web workers in your use-case. Note that the copy is avoided if and only if the environment supports transferable objects (see https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Transferable_objects) in web worker messages.