Restructure guest/host error handling (hyperlight-dev#868)

* Add test that fails due memory leaking when host function error occurs

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Update FlatBuffer schema and generation process

- Add all.fbs to include all schema files in one place
- Restructure function_call_result.fbs to use Result-like union
- Add HostError variant to ErrorCode enum in guest_error.fbs
- Update flatbuffer generation command in Justfile to use all.fbs
- Update documentation for new generation process

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Regenerate FlatBuffer generated code

Update all generated Rust code based on the new schema definitions.
This includes new types for error handling and result structures.

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Refactor error handling types and utilities

- Update function_types.rs to handle Result-like return values
- Simplify guest_error.rs wrapper implementation
- Update util.rs for new generated types
- Update mod.rs  for new generated types

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Remove obsolete guest error handling

- Remove guest_err.rs from hyperlight_host (replaced by new error handling)
- Remove guest_err.rs from hyperlight_guest_bin (replaced by new error handling)
- Update func/mod.rs to remove obsolete import

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Update host-side error handling

- Update initialized_multi_use.rs to use new Result-like error handling
- Update mem/mgr.rs to handle host function errors properly
- Update sandbox/outb.rs for new error propagation pattern

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Update guest-side error handling

- Update guest/host_comm.rs to use new Result-like return values
- Update guest_bin/call.rs to properly handle host function errors
- Update guest_bin/lib.rs to remove obsolete error handling import and make GUEST_HANDLE public (for use in C-API)
- Update guest_capi/error.rs to support new error types

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Fix up test to work with new error handling

Update sandbox_host_tests.rs to use the new Result-like error handling pattern.

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Update Cargo dependencies

Update Cargo.lock and Cargo.toml files to reflect the dependency changes
needed for the new error handling implementation.

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Fix test so it passes

Signed-off-by: Ludvig Liljenberg <[email protected]>

* First round of PR feedback

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Unignore forgotten test

Signed-off-by: Ludvig Liljenberg <[email protected]>

* Update flatc version in docs

Signed-off-by: Ludvig Liljenberg <[email protected]>

---------

Signed-off-by: Ludvig Liljenberg <[email protected]>

Author: ludfjig

Cargo.lock

@@ -372,7 +372,7 @@ build-fuzzer fuzz-target:
 ###################
 
 gen-all-fbs-rust-code:
-    for fbs in `find src -name "*.fbs"`; do flatc -r --rust-module-root-file --gen-all -o ./src/hyperlight_common/src/flatbuffers/ $fbs; done
+    flatc --rust --rust-module-root-file --gen-all -o ./src/hyperlight_common/src/flatbuffers/ ./src/schema/all.fbs
     just fmt-apply
 
 install-vcpkg:

Justfile

@@ -1,6 +1,6 @@
 # How to use FlatBuffers
 
-> Note: the last generation of the flatbuffer code was with done with flatc version 25.2.10 (i.e., the last version as of May 1st, 2025).
+> Note: the last generation of the flatbuffer code was with done with flatc version 25.9.23 (i.e., the last version as of Oct 2nd, 2025).
 
 Flatbuffers is used to serialize and deserialize some data structures.
 
@@ -19,9 +19,3 @@ We recommend building `flatc` from source. To generate rust code, use
 ```console
 just gen-all-fbs-rust-code
 ```
-
-### Note about generated code
-
-Because we invoke `flatc` multiple times when generating the Rust code, the `mod.rs` generated in `./src/hyperlight_common/src/flatbuffers` is overwritten multiple times and will likely be incorrect. Make sure to manually inspect and if necessary update this file before continuing with your changes as certain modules might be missing. After fixing `mod.rs`, you might need to re-run `just fmt`, since it might not have applied to all generated files if your `mod.rs` was invalid.
-
->`flatc` does support passing multiple schema files (e.g. it is possible to pass `.\src\schema\*.fbs`), so we could regenerate all the files each time a change was made, however that generates incorrect code (see [here](https://github.com/google/flatbuffers/issues/6800) for details).

docs/how-to-use-flatbuffers.md

@@ -17,16 +17,11 @@ limitations under the License.
 extern crate flatbuffers;
 
 use alloc::string::{String, ToString};
-use alloc::vec::Vec;
 
-use anyhow::{Error, Result};
-use flatbuffers::size_prefixed_root;
 #[cfg(feature = "tracing")]
 use tracing::{Span, instrument};
 
-use crate::flatbuffers::hyperlight::generated::{
-    ErrorCode as FbErrorCode, GuestError as FbGuestError, GuestErrorArgs,
-};
+use crate::flatbuffers::hyperlight::generated::ErrorCode as FbErrorCode;
 
 #[derive(Debug, Clone, Copy, Eq, PartialEq)]
 #[repr(C)]
@@ -48,6 +43,7 @@ pub enum ErrorCode {
     GuestFunctionParameterTypeMismatch = 14,
     GuestError = 15,
     ArrayLengthParamIsMissing = 16,
+    HostFunctionError = 17,
 }
 
 impl From<ErrorCode> for FbErrorCode {
@@ -73,6 +69,7 @@ impl From<ErrorCode> for FbErrorCode {
             }
             ErrorCode::GuestError => Self::GuestError,
             ErrorCode::ArrayLengthParamIsMissing => Self::ArrayLengthParamIsMissing,
+            ErrorCode::HostFunctionError => Self::HostError,
         }
     }
 }
@@ -99,6 +96,7 @@ impl From<FbErrorCode> for ErrorCode {
             }
             FbErrorCode::GuestError => Self::GuestError,
             FbErrorCode::ArrayLengthParamIsMissing => Self::ArrayLengthParamIsMissing,
+            FbErrorCode::HostError => Self::HostFunctionError,
             _ => Self::UnknownError,
         }
     }
@@ -123,6 +121,7 @@ impl From<u64> for ErrorCode {
             14 => Self::GuestFunctionParameterTypeMismatch,
             15 => Self::GuestError,
             16 => Self::ArrayLengthParamIsMissing,
+            17 => Self::HostFunctionError,
             _ => Self::UnknownError,
         }
     }
@@ -147,6 +146,7 @@ impl From<ErrorCode> for u64 {
             ErrorCode::GuestFunctionParameterTypeMismatch => 14,
             ErrorCode::GuestError => 15,
             ErrorCode::ArrayLengthParamIsMissing => 16,
+            ErrorCode::HostFunctionError => 17,
         }
     }
 }
@@ -174,6 +174,7 @@ impl From<ErrorCode> for String {
             }
             ErrorCode::GuestError => "GuestError".to_string(),
             ErrorCode::ArrayLengthParamIsMissing => "ArrayLengthParamIsMissing".to_string(),
+            ErrorCode::HostFunctionError => "HostFunctionError".to_string(),
         }
     }
 }
@@ -194,44 +195,6 @@ impl GuestError {
     }
 }
 
-impl TryFrom<&[u8]> for GuestError {
-    type Error = Error;
-    fn try_from(value: &[u8]) -> Result<Self> {
-        let guest_error_fb = size_prefixed_root::<FbGuestError>(value)
-            .map_err(|e| anyhow::anyhow!("Error while reading GuestError: {:?}", e))?;
-        let code = guest_error_fb.code();
-        let message = match guest_error_fb.message() {
-            Some(message) => message.to_string(),
-            None => String::new(),
-        };
-        Ok(Self {
-            code: code.into(),
-            message,
-        })
-    }
-}
-
-impl TryFrom<&GuestError> for Vec<u8> {
-    type Error = Error;
-    #[cfg_attr(feature = "tracing", instrument(err(Debug), skip_all, parent = Span::current(), level= "Trace"))]
-    fn try_from(value: &GuestError) -> Result<Vec<u8>> {
-        let mut builder = flatbuffers::FlatBufferBuilder::new();
-        let message = builder.create_string(&value.message);
-
-        let guest_error_fb = FbGuestError::create(
-            &mut builder,
-            &GuestErrorArgs {
-                code: value.code.into(),
-                message: Some(message),
-            },
-        );
-        builder.finish_size_prefixed(guest_error_fb, None);
-        let res = builder.finished_data().to_vec();
-
-        Ok(res)
-    }
-}
-
 impl Default for GuestError {
     #[cfg_attr(feature = "tracing", instrument(parent = Span::current(), level= "Trace"))]
     fn default() -> Self {