Fixes FP rounding issue and adds tests

[jsturtevant]

fixes #86

The issue was caused because cranelift compilation assume SSE2 even for the where as rust compiler doesn't include these instructions by default. This means when transitioning through wasmtime libcalls the parameters are lost since wasm is using SSE2 instructions and wasmtime isn't. The more advance SSE instructions require a separate pr in HL core that is needed to enable them.

[ludfjig]

Nice catch 👍 LGTM. Did you check that the tests fail without the -soft-float,+sse,+sse2?

[ludfjig]

Btw, I am not sure if you already have, but we should double check that we set all required cpu control registers for enabling sse,sse2. I know x86_64 assumes sse,sse2 are part base ISA, but still might need some setup. In addition, we should make sure these are reset between guest functions call to avoid any information leaks (if they aren't already).

[jsturtevant]

Nice catch 👍 LGTM. Did you check that the tests fail without the -soft-float,+sse,+sse2?

Yes. Also tested this against the sample in #86 (comment)

Btw, I am not sure if you already have, but we should double check that we set all required cpu control registers for enabling sse,sse2. I know x86_64 assumes sse,sse2 are part base ISA, but still might need some setup. In addition, we should make sure these are reset between guest functions call to avoid any information leaks (if they aren't already).

These seem to be enabled in KVM (I check the mxcsr register). I haven't verified on Windows/MSVH. Maybe the best thing do here it push hyperlight-dev/hyperlight#686 forward as well.

Good call on clearing registers which will need to happen in HL, will work on a change for that.

[jsturtevant]

updated this to use hyperlight-dev/cargo-hyperlight#1 builds the entire toolchain properly.

[ludfjig]

Should we switch hyperlight core to use this target for its testing before merging this? I think it would be valuable

[jsturtevant]

Should we switch hyperlight core to use this target for its testing before merging this? I think it would be valuable

Yes that's the plan. wanted to start with the basics where I knew it would fix an issue. I am running into some issues though here though.

[jsturtevant]

The stack overflow error is coming from 68deb95 these patches work at d2da14e

[jsturtevant]

The stack overflow error is coming from 68deb95 these patches work at d2da14e

The tracing instrument was running in the exception context, it's fairly complicated and does save a few things to xmm register with the change of the FP ABI. since the exception handler doesn't guarantee the stack is aligned when the instrument macro was saving the xmm values it caused an GP fault exception. There was an instrumentation on the outb function and this caused it to thrash handling exceptions until it stack overflowed.

[jsturtevant]

A couple more notes:

• this didn't trigger in debug mode because optimizations that used the xmm registers now that they are available (via the new target) were not enabled.
• We might not want run the tracing code on an interrupt like we are doing here. There is some work in progress that could fix this but it was useful to find this bug in our exception handling

The fix for this being worked on in hyperlight-dev/hyperlight#1037

[jsturtevant]

Should we switch hyperlight core to use this target for its testing before merging this? I think it would be valuable

Yes that's the plan. wanted to start with the basics where I knew it would fix an issue. I am running into some issues though here though.

done here: hyperlight-dev/hyperlight#1084