Issues updating to latest WASI SDK

[radu-matei]

I'm in the process of updating this to use the latest WASI SDK for .NET — https://www.nuget.org/packages/Wasi.Sdk/0.1.2-preview.10061

Bumped the package version in both the sample and this SDK:

<PackageReference Include="Wasi.Sdk" Version="0.1.2-preview.10061" />

However, I am seeing two distinct problems:

• when UseWizer is enabled, pre-initialising the module fails with the following error related to the bulk memory proposal not enabled

  Running wizer to preinitialize bin/Debug/net7.0/HelloWorld.Spin.wasm...
EXEC : error : failed to compile the Wasm module [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/samples/hello-world/HelloWorld.Spin.csproj]

  Caused by:
      0: WebAssembly translation error
      1: Invalid input WebAssembly code at offset 4252388: bulk memory support is not enabled
/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/build/Fermyon.Spin.Sdk.targets(18,3): error MSB3073: The command "wizer bin/Debug/net7.0/HelloWorld.Spin.wasm -o bin/Debug/net7.0/HelloWorld.Spin.wasm.pre.wasm --allow-wasi" exited with code 1. [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/samples/hello-world/HelloWorld.Spin.csproj]

Using a branch of Wizer that enables the bulk memory proposal fixes this issue — however, that is not currently enabled in Wizer, nor is it configurable. Depending on the cause of this issue, we should consider adding an optional configuration to Wizer to enable this proposal.

Note that enabling Wizer on a simple WASI console application from C# does not encounter this issue.

• running a release build results in a linking error:

wasm-ld : error : /var/folders/rb/3k3_t3zd0j32fh_b7kx54rq80000gn/T/dotnet_wasi_getbundledfile-6d21e9.o: undefined symbol: Fermyon_Spin_Sdk_dll_B1FE6C56_o_len [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]
wasm-ld : error : /var/folders/rb/3k3_t3zd0j32fh_b7kx54rq80000gn/T/dotnet_wasi_getbundledfile-6d21e9.o: undefined symbol: Fermyon_Spin_Sdk_dll_B1FE6C56_o [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]
clang-14 : error : linker command failed with exit code 1 (use -v to see invocation) [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]
/Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/Wasi.Sdk.targets(224,3): error MSB3073: The command ""/Users/radu/.wasi-sdk/wasi-sdk-16.0/bin/clang" /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../native/main.c /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-ee-interp.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-ilgen.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-wasi-driver.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmonosgen-2.0.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-hot_reload-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-marshal-ilgen-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-diagnostics_tracing-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libmono-component-debugger-stub-static.a /Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/libSystem.Native.a obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/dotnet_wasi_getbundledfile.c obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/Fermyon.Spin.Sdk.dll.B1FE6C56.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Net.Http.dll.BFD8CFD7.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Net.Security.dll.4C5B4FF4.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.dll.CB61E222.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Private.CoreLib.dll.CA1E5D08.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Memory.dll.4CE36FB8.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Security.Cryptography.dll.F320C359.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.NonGeneric.dll.DE2976E7.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.dll.7D26D3DA.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.dll.F876D186.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Private.Uri.dll.EA8A7CD0.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Net.Primitives.dll.1E0BF110.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/Microsoft.Win32.Primitives.dll.7502414D.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Diagnostics.Tracing.dll.2755647A.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.InteropServices.dll.D3ADDD17.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Text.Encoding.Extensions.dll.B1243078.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.Concurrent.dll.D9EB7C68.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.Numerics.dll.404B5944.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Formats.Asn1.dll.923319B4.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Diagnostics.DiagnosticSource.dll.D9275F9C.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.Thread.dll.59374F80.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Runtime.InteropServices.JavaScript.dll.6EAC5B7A.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.Channels.dll.2396D734.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Threading.ThreadPool.dll.1C2B1053.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Collections.Immutable.dll.FFD8A487.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Linq.dll.78C0A655.o obj/Release/net7.0/dotnet-wasi-sdk/bundle-objects/System.Numerics.Vectors.dll.D9C4D162.o obj/Release/net7.0/dotnet-wasi-sdk/entrypoint_Fermyon.Spin.Sdk.c --sysroot="/Users/radu/.wasi-sdk/wasi-sdk-16.0/share/wasi-sysroot" -I"/Users/radu/.nuget/packages/wasi.sdk/0.1.2-preview.10061/build/../packs/wasi-wasm/native/include" -Wl,--export=malloc,--export=free,--export=__heap_base,--export=__data_end -Wl,-z,stack-size=1048576,--initial-memory=52428800,--max-memory=524288000,-lwasi-emulated-mman -Wl,-s -o "obj/Release/net7.0/dotnet-wasi-sdk/Fermyon.Spin.Sdk.wasm"" exited with code 1. [/Users/radu/projects/src/github.com/fermyon/spin-dotnet-sdk/src/Fermyon.Spin.Sdk.csproj]

Note that running a release build on a simple WASI console application from C# does not encounter this issue.

cc @itowlson @SteveSandersonMS

[radu-matei]

It turns out that while just enabling support for the bulk memory proposal is enough to apparently solve the issue, it is not something
It turns out just enabling support for bulk memory in Wizer is not trivial, at least in order to have robust support — see bytecodealliance/wizer#37 (review)

[SteveSandersonMS]

For the issue about undefined symbol: Fermyon_Spin_Sdk_dll_B1FE6C56_o_len, could you delete the bin/obj directories from your local copy (or maybe run git clean -xdf) and try again?

There was an issue in the older SDK whereby it didn't know how to be properly incremental for those files, which I think is fixed in the updated SDK. But I didn't test that the updated SDK knows how to deal with the temporary files left behind by an older SDK.

If that doesn't work, please let me know.

[radu-matei]

🤦

Ok, I'm super embarrassed about this... both issues are now solved... thanks!

[radu-matei]

Well, it turns out this is still an issue.
I just re-installed Wizer (had a patched version locally that enabled bulk memory operations) and I (and @itowlson) am still seeing the same issue:

  Running wizer to preinitialize bin/Release/net7.0/HelloWorld.Spin.wasm...
EXEC : error : failed to compile the Wasm module [/github.com/fermyon/spin-dotnet-sdk/samples/hello-world/HelloWorld.Spin.csproj]

  Caused by:
      0: WebAssembly translation error
      1: Invalid input WebAssembly code at offset 4252388: bulk memory support is not enabled
/github.com/fermyon/spin-dotnet-sdk/src/build/Fermyon.Spin.Sdk.targets(18,3): error MSB3073: The command "wizer bin/Release/net7.0/HelloWorld.Spin.wasm -o bin/Release/net7.0/HelloWorld.Spin.wasm.pre.wasm --allow-wasi" exited with code 1. [/github.com/fermyon/spin-dotnet-sdk/samples/hello-world/HelloWorld.Spin.csproj]

[itowlson]

It looks like with Wasi.Sdk 1.2, the memcpy instructions generated by the wit-bindgen c get compiled to memory.copy. I haven't been able to find the exact same piece of code when compiled with 1.1, but memory.copy appears nowhere in 1.1 output. I assume 1.1 is compiling memcpy to a loop.

memcpy appears in the source code only as part of the various generated blah_string_dup routines. So we can probably work around this by postprocessing the generated files. We already have to manually tweak the include so could add a #define for memcpy while we're at it.

My EXTREMELY UNTESTED guess is that this starting happening when Wasi.Sdk upgraded from WASI SDK 12 to 16, in SteveSandersonMS/dotnet-wasi-sdk@4fefe57. There's no change in the arguments so I AGAIN EXTREMELY UNTESTEDLY guess this might be a change in WASI SDK defaults?

[itowlson]

Aha

[radu-matei]

So if we want to use Wizer for things, the WASI .NET SDK needs to downgrade to WASI SDK 15?

[itowlson]

I'm trying to figure out if there's a compiler flag to use loops instead of bulk memory, but am having no luck with the docs - do you know a good reference?

If such a flag exists, we could request that Wasi.Sdk provides an option for consumers to set custom compiler options on the Clang invocation. Then consumers that didn't care about Wizer - such as ASP.NET applications that are long lived and amortise their startup cost - could use the fast operations, and consumers like us who cared more about fast startup and didn't need much string copying could drop back to Wizerable ones.

If no such flag exists, our options are:

• Petition Wasi.Sdk to stay on WASI SDK 15 until Wizer updates.
• Substitute our own implementation of memcpy when Wizering.

(unless I am missing an alternative of course!)

[itowlson]

Ah, looks like there is a -mno-bulk-memory option. So if Wasi.Sdk provided a way to inject that, we would be gold.

[itowlson]

Raised SteveSandersonMS/dotnet-wasi-sdk#33 for discussion.

[dicej]

FYI, I'm aiming to post a PR to allow memory.fill, memory.copy, and memory.init operations in Wizer, which should hopefully cover the SDK's use of bulk memory ops.

[dicej]

bytecodealliance/wizer#57