Issue of wasm memory limit of 2G

[0xsyj]

When I was trying to build the AST of a relatively large source code file, I encountered an error:

[-] new Parser and parse failed, try again.
RuntimeError: memory access out of bounds
RuntimeError: memory access out of bounds
    at wasm://wasm/001db6c6:wasm-function[586]:0x20678
    at wasm://wasm/001db6c6:wasm-function[440]:0x1358c
    at wasm://wasm/001db6c6:wasm-function[80]:0x2565
    at wasm://wasm/001db6c6:wasm-function[52]:0x1bb8
    at wasm://wasm/001db6c6:wasm-function[1873]:0x67913
    at wasm://wasm/001db6c6:wasm-function[1874]:0x67a36
    at dynCall (D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:904:37)
    at D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:907:49
    at Object.createUnit (D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:1080:16)
    at new _Parser (D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:11389:22)

Later, I checked node_modules\cxx-frontend\dist\cjs\index.cjs and found that the failure occurred at this line:

var base = _malloc(4 + length + 1);

Memory allocation failed. By checking wasmMemory.buffer.byteLength, I noticed that it could not grow beyond 2GB.

After some research, I found the reason: https://emscripten.org/docs/tools_reference/settings_reference.html#maximum-memory. The default maximum memory for WebAssembly is 2GB. This issue can be resolved by specifying ALLOW_MEMORY_GROWTH=1 and MAXIMUM_MEMORY=4GB during the compilation of the WebAssembly module.

I proceeded to recompile the WebAssembly module in the project with these settings. However, I found that the memory was still insufficient because the source code is indeed very large (some open-source projects).

Later, I discovered https://emscripten.org/docs/tools_reference/settings_reference.html#memory64, which allows setting memory beyond 4GB by enabling memory64=1 during compilation and linking. However, when I ran the compiled code, I encountered some strange errors that I couldn't resolve. For example, in the compiled index.cjs, the line wasmTable.get(BigInt(funcPtr)) threw an error: Cannot convert a BigInt value to a number. After converting it to a number, I continued running the code, but then encountered another error: table index out of bounds.

It seems that the project was not designed with the consideration of requiring such a large amount of memory. I hope the development team can address this issue. Thanks.

[robertoraggi]

Thanks for the report. I’m aware of some high memory usage during the preprocessing stage. I plan to rework the preprocessor after I’ve finished with the binder and template instantiation. I test the parser on some large projects that interest me. If you could share the Open Source project that fail to parse I can have a look.

[0xsyj]

Just like this c source file, it only has 601KB, but it will directly report an error when obtaining its AST.（I've expanded it to 4G memory limit.）
libtomcrypt.zip

RuntimeError: memory access out of bounds
RuntimeError: memory access out of bounds
at wasm://wasm/001db6c6:wasm-function[47]:0x17ba
at wasm://wasm/001db6c6:wasm-function[999]:0x4695a
at wasm://wasm/001db6c6:wasm-function[1873]:0x6796e
at wasm://wasm/001db6c6:wasm-function[1874]:0x67a36
at dynCall (D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:904:37)
at D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:907:49
at Object.createUnit (D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:1080:16)
at new _Parser (D:\OneDrive\CodeSource\node_modules\cxx-frontend\dist\cjs\index.cjs:11389:22)

[0xsyj]

Can you let the project support wasm-memory64 so that the memory can be unlimited.
https://emscripten.org/docs/tools_reference/settings_reference.html#memory64

[0xsyj]

I use child_process.fork,https://nodejs.org/api/child_process.html#child_processforkmodulepath-args-options , So that parsing each source code and obtaining AST becomes a separate process with a separate v8 engine, but once the source code size is about 500kb, 4G is not enough when obtaining AST.

[robertoraggi]

Thanks for sharing the code. It appears to be a bug in the parser that causes an infinite loop during the parsing of the der_encode_utctime function. The casts inside that function cause the error, e.g.

 out[x++] = der_ia5_char_encode(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(((char*)(Kmneefofclic((char*)baseten, 10)))))))))))))))))))))))))))))))))[(utctime->YY / 10) % 10]);

when I remove such casts from the function then it seems to parse fine.

node build.em/src/frontend/cxx.js  -D_WASI_EMULATED_SIGNAL -D__cdecl="" libtomcrypt.c

I will try to fix it soon. Thanks again for the report.

[robertoraggi]

I’ve made a potential fix in #496

If you’d like to test it, you can paste the file’s content into this test editor

https://robertoraggi.github.io/cplusplus/?path=/story/cxxfrontend-syntaxtree--basic

[0xsyj]

I just said that 500kb will not take up so much memory. Thank you for the fix, I'll try it

[robertoraggi]

the problem with memory64 is it is not widely used yet, It may be a little bit too early to use it.

That said, I do test with larger payloads this file expands to 5.6mb after preprocessing, node 23.9 seems to be fine.
I also test the parser by parsing the entire llvm codebase, that also seems to work okay. Nevertheless I will try to reduce the memory consumption.

node build.em/src/frontend/cxx.js -I /opt/homebrew/opt/boost/include /opt/homebrew/opt/boost/include/boost/graph/adjacency_list.hpp -E | wc -c
 5575266

node build.em/src/frontend/cxx.js -I /opt/homebrew/opt/boost/include /opt/homebrew/opt/boost/include/boost/graph/adjacency_list.hpp -ast-dump