Update FlatBuffer schema and generation process

- Add all.fbs to include all schema files in one place
- Restructure function_call_result.fbs to use Result-like union
- Add HostError variant to ErrorCode enum in guest_error.fbs
- Update flatbuffer generation command in Justfile to use all.fbs
- Update documentation for new generation process

Signed-off-by: Ludvig Liljenberg <[email protected]>

Author: ludfjig

Justfile

@@ -359,7 +359,7 @@ build-fuzzer fuzz-target:
 ###################
 
 gen-all-fbs-rust-code:
-    for fbs in `find src -name "*.fbs"`; do flatc -r --rust-module-root-file --gen-all -o ./src/hyperlight_common/src/flatbuffers/ $fbs; done
+    flatc --rust --rust-module-root-file --gen-all -o ./src/hyperlight_common/src/flatbuffers/ ./src/schema/all.fbs
     just fmt-apply
 
 install-vcpkg:

docs/how-to-use-flatbuffers.md

@@ -19,9 +19,3 @@ We recommend building `flatc` from source. To generate rust code, use
 ```console
 just gen-all-fbs-rust-code
 ```
-
-### Note about generated code
-
-Because we invoke `flatc` multiple times when generating the Rust code, the `mod.rs` generated in `./src/hyperlight_common/src/flatbuffers` is overwritten multiple times and will likely be incorrect. Make sure to manually inspect and if necessary update this file before continuing with your changes as certain modules might be missing. After fixing `mod.rs`, you might need to re-run `just fmt`, since it might not have applied to all generated files if your `mod.rs` was invalid.
-
->`flatc` does support passing multiple schema files (e.g. it is possible to pass `.\src\schema\*.fbs`), so we could regenerate all the files each time a change was made, however that generates incorrect code (see [here](https://github.com/google/flatbuffers/issues/6800) for details).

src/schema/all.fbs

@@ -0,0 +1,7 @@
+include "function_types.fbs";
+include "function_call_result.fbs";
+include "function_call.fbs";
+include "guest_error.fbs";
+include "guest_log_data.fbs";
+include "host_function_definition.fbs";
+include "host_function_details.fbs";

src/schema/function_call_result.fbs

@@ -1,9 +1,21 @@
 include "function_types.fbs";
+include "guest_error.fbs";
 
 namespace Hyperlight.Generated;
 
+// Wrapper so ReturnValue (a union) can be a single union variant
+table ReturnValueBox {
+    value: ReturnValue (required);
+}
+
+// Result-like union
+union FunctionCallResultType {
+    ReturnValueBox,
+    GuestError
+}
+
 table FunctionCallResult {
-    return_value:ReturnValue(required);
+    result: FunctionCallResultType (required);
 }
 
 root_type FunctionCallResult;

src/schema/guest_error.fbs

@@ -16,7 +16,8 @@ enum  ErrorCode: ulong {
     MallocFailed = 13,                              // this error is set when malloc returns 0 bytes.
     GuestFunctionParameterTypeMismatch =    14,     // The function call parameter type was not the expected type.  
     GuestError  = 15,                               // An error occurred in the guest Guest implementation should use this along with a message when calling setError.
-    ArrayLengthParamIsMissing = 16                  // Expected a int parameter to follow a byte array
+    ArrayLengthParamIsMissing = 16,                 // Expected a int parameter to follow a byte array
+    HostError = 17                                  // Guest called Host Function, which errored.
 }
 
 table GuestError {