Skip to content

Improve Compression from Generator #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zalo wants to merge 4 commits into from
Closed

Improve Compression from Generator #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f75af4d
Improve Binary Serialization in the Code Generator
zalo May 8, 2025
0965e29
Fix silly gaps
zalo May 8, 2025
8f6142a
Add Compression to Code Generator
zalo May 8, 2025
8deaa48
Merge branch 'main' into compressed-generator
zalo May 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions pages/editor.vue
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,12 +58,6 @@ definePageMeta({
<input type="checkbox" id="offline">
<span class="checkmark2"></span>
</label>
<ul>
<label class="container" id="singleFile" style="display:none;">Pack into single file
<input type="checkbox" id="offlinePack">
<span class="checkmark2"></span>
</label>
</ul>
<div id="pathToData">
<label class="container">Use Local path to Data
<input type="checkbox" id="p2d">
Expand Down
42 changes: 20 additions & 22 deletions public/js/main.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ function editorMain() {
let gameName = (document.getElementById('nameOfGame').value && document.getElementById('nameOfGame').value.trim() !== '') ? document.getElementById('nameOfGame').value : file.name;
try {
//let isChromebook = window.navigator.userAgent.includes("CrOS");
//window.sendLog("A user just generated a page with the EmulatorJS code helper.\n Game Name: "+gameName+"\n File name: "+file.name+"\n System: "+window.selectedCoreData.core+"\n Offline: "+(offline.checked ? (document.getElementById('offlinePack').checked ? "pack" : "yes") : "no")+"\n Custom path to data: "+((document.getElementById('p2d').checked&&path2Data.value.trim()) ? path2Data.value : "no")+"\n Ad Url: "+(document.getElementById('adUrl').value.trim() ? "<"+document.getElementById('adUrl').value.trim()+">" : "no")+"\n Is Chromebook: "+(isChromebook ? "yes" : "no"));
//window.sendLog("A user just generated a page with the EmulatorJS code helper.\n Game Name: "+gameName+"\n File name: "+file.name+"\n System: "+window.selectedCoreData.core+"\n Offline: "+(offline.checked ? "pack" : "no")+"\n Custom path to data: "+((document.getElementById('p2d').checked&&path2Data.value.trim()) ? path2Data.value : "no")+"\n Ad Url: "+(document.getElementById('adUrl').value.trim() ? "<"+document.getElementById('adUrl').value.trim()+">" : "no")+"\n Is Chromebook: "+(isChromebook ? "yes" : "no"));
} catch (e) { }
data['EJS_player'] = '#game';
data['EJS_core'] = window.selectedCoreData.core;
Expand Down Expand Up @@ -136,7 +136,7 @@ function editorMain() {
}

let zipOut = true;
let fileData = '<html>\n <head>\n <!--HTML file auto generated using EmulatorJS codehelper-->\n <style>\n body, html {\n margin: 0;\n padding: 0;\n }\n </style>\n </head>\n <body>\n <div style="width:100%;height:100%;max-width:100%">\n <div id="game"></div>\n </div>\n <script>\n';
let fileData = '<html>\n <head>\n <!--HTML file auto generated using EmulatorJS codehelper-->\n <style>\n body, html {\n margin: 0;\n padding: 0;\n }\n </style>\n </head>\n <body>\n <div style="width:100%;height:100%;max-width:100%">\n <div id="game"></div>\n </div>\n <script type="module">\n';
const spaces = ' ';
if (!offline.checked) {
if (stateOnLoad.files[0]) {
Expand All @@ -151,38 +151,30 @@ function editorMain() {
}
for (var k in data) {
if (data[k] === true || data[k] === false) {
fileData += (spaces + k + ' = ' + data[k] + ';\n');
fileData += (spaces + "window." + k + ' = ' + data[k] + ';\n');
} else {
fileData += (spaces + k + ' = "' + data[k] + '";\n');
fileData += (spaces + "window." + k + ' = "' + data[k] + '";\n');
}
}
fileData += ' </script>\n <script src="' + data['EJS_pathtodata'] + 'loader.js"></script>\n </body>\n</html>';
} else if (document.getElementById('offlinePack').checked) {
data['EJS_gameUrl'] = 'new Blob([Uint8Array.from(atob(window.gameData), (m) => m.codePointAt(0))])';
var b = bytesToBase64(new Uint8Array(await (new Blob([file])).arrayBuffer()));
} else {
data['EJS_gameUrl'] = 'await new Response(new Blob([Uint8Array.from(atob(window.gameData), (m) => m.codePointAt(0))]).stream().pipeThrough(new DecompressionStream("gzip"))).blob()';
var b = bytesToBase64(new Uint8Array(await (await CompressBlob(new Blob([file]))).arrayBuffer()));
var a = spaces + 'window.gameData = `' + b + '`;\n';
fileData += a;
for (var k in data) {
if (data[k] === true || data[k] === false || k === 'EJS_gameUrl') {
fileData += (spaces + k + ' = ' + data[k] + ';\n');
fileData += (spaces + "window." + k + ' = ' + data[k] + ';\n');
} else {
fileData += (spaces + k + ' = "' + data[k] + '";\n');
fileData += (spaces + "window." + k + ' = "' + data[k] + '";\n');
}
}
fileData += ' </script>\n <script src="' + data['EJS_pathtodata'] + 'loader.js"></script>\n </body>\n</html>';
fileData += ` let emulatorScript = document.createElement("script");
emulatorScript.type = 'text/javascript';
emulatorScript.src = '`+data['EJS_pathtodata']+`loader.js';
document.body.appendChild(emulatorScript);\n`
fileData += ' </script>\n </body>\n</html>';
zipOut = false;
} else {
data['EJS_gameUrl'] = 'new Blob([Uint8Array.from(atob(window.gameData), (m) => m.codePointAt(0))])';
var b = bytesToBase64(new Uint8Array(await (new Blob([file])).arrayBuffer()));
zip.file('gameData.js', 'window.gameData = `' + b + '`\n');
for (var k in data) {
if (data[k] === true || data[k] === false || k === 'EJS_gameUrl') {
fileData += (spaces + k + ' = ' + data[k] + ';\n');
} else {
fileData += (spaces + k + ' = \'' + data[k] + '\';\n');
}
}
fileData += ' </script>\n <script src=\'' + data['EJS_pathtodata'] + 'loader.js\'></script>\n </body>\n</html>';
}
if (zipOut === false) {
document.getElementById('select2').style = 'display:none;';
Expand Down Expand Up @@ -272,3 +264,9 @@ function bytesToBase64(bytes) {
).join("");
return btoa(binString);
}

async function CompressBlob(blob) {
const cs = new CompressionStream("gzip");
const compressedStream = blob.stream().pipeThrough(cs);
return await new Response(compressedStream).blob();
}
Comment on lines +268 to +272
Copy link
Member

@ethanaobrien ethanaobrien May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compression using CompressionStream should fallback if its unsupported, since this api is still relatively new

Copy link
Contributor Author

@zalo zalo May 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API#browser_compatibility
It looks like most browsers got it in 2020, and the stragglers got it in 2023... so I think the odds are pretty low...

Since it's also used on the decompression side... should the baked .html outputs optionally import pako if it can't find Compression/Decompression Streams?

Copy link
Member

@ethanaobrien ethanaobrien May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the odds are pretty low

You'd be surprised :). I had several issues a while back (a few months ago) when I broke support for a GPU that hasn't had updates for 5 years so it's definitely needed.

I would rather not pull in dependencies with the output code, though if you relied on zip compression then EmulatorJS does have the ability to detect and decompress that

Copy link
Contributor Author

@zalo zalo May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I can switch the compression for deflate, and check to see if EmulatorJS can handle that directly... is there anything I need to do to let it know that it's compressed? Perhaps a MIME type in the blob URL?

Copy link
Contributor Author

@zalo zalo May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bah, CompressionStreams can natively produce .gz files, but not .zip files (which can be understood by EmulatorJS).

Luckily, it turns out there's a way to make JSZip actually run compression 😅 https://stackoverflow.com/a/41614764
Hopefully it's not too much slower... May want to try littlezipper if it is...

So perhaps I'll discard this PR in exchange for using the existing libraries...