feat: msgpack encoding for Program and WitnessStack

barretenberg/cpp/cmake/msgpack.cmake

@@ -13,4 +13,4 @@ ExternalProject_Add(
     BUILD_COMMAND ""      # No build step
     INSTALL_COMMAND ""    # No install step
     UPDATE_COMMAND ""     # No update step
-)
+)

barretenberg/cpp/src/barretenberg/dsl/README.md

@@ -8,14 +8,19 @@ There are two types of breaking serialization changes. One that alters the inter
 
 1. Internal Structure Change
 
-Go to the ACVM `acir` crate and re-run the [serde reflection test](https://github.com/noir-lang/noir/blob/master/acvm-repo/acir/src/lib.rs#L51). Remember to comment out the hash check to write the updated C++ serde file. Copy that file into the `dsl` package's [serde folder](./acir_format/serde/) where you will see an `acir.hpp` file.
-
-You will have to update a couple things in the new `acir.hpp`:
-
-- Replace all `throw serde::deserialization_error` with `throw_or_abort`
-- The top-level struct (such as `Program`) will still use its own namespace for its internal fields. This extra `Program::` can be removed from the top-level `struct Program { .. }` object.
-
-The same can then be done for any breaking changes introduced to `witness_stack.hpp`.
+The C++ bindings are generated by the ACVM `acir` crate and while running the [serde reflection test](https://github.com/noir-lang/noir/blob/master/acvm-repo/acir/src/lib.rs#L51).
+By default it just ascertains that the format has not changed, but it can be told to
+write the updated mappings to `acir.cpp` and `witness_stack.cpp`,
+which become `acir.hpp` and `witness_stack.hpp` in `aztec-packages`.
+
+The code can be regenerated and copied over with the following commands:
+
+```shell
+cd noir/noir-repo && NOIR_CODEGEN_OVERWRITE=1 cargo test -p acir cpp_codegen && cd -
+cp noir/noir-repo/acvm-repo/acir/codegen/acir.cpp barretenberg/cpp/src/barretenberg/dsl/acir_format/serde/acir.hpp
+cp noir/noir-repo/acvm-repo/acir/codegen/witness.cpp barretenberg/cpp/src/barretenberg/dsl/acir_format/serde/witness_stack.hpp
+cd barretenberg/cpp && ./format.sh changed && cd -
+```
 
 2. Full Breaking Change
 

barretenberg/cpp/src/barretenberg/dsl/acir_format/acir_to_constraint_buf.hpp

@@ -4,18 +4,43 @@
 
 namespace acir_format {
 
-AcirFormat circuit_buf_to_acir_format(std::vector<uint8_t> const& buf, uint32_t honk_recursion);
+/**
+ * @brief Deserialize `buf` either based on the first byte interpreted as a
+          Noir serialization format byte, or falling back to `bincode` if
+          the format cannot be recognized. Currently only `msgpack` format
+          is expected, or the legacy `bincode` format.
+ * @note Due to the lack of exception handling available to us in Wasm we can't
+ *       try `bincode` format and if it fails try `msgpack`; instead we have to
+ *       make a decision and commit to it.
+ */
+template <typename T>
+T deserialize_any_format(std::vector<uint8_t> const& buf,
+                         std::function<T(msgpack::object const&)> decode_msgpack,
+                         std::function<T(std::vector<uint8_t>)> decode_binpack);
+
+/**
+ * @brief Deserializes a `Program` from bytes, trying `msgpack` or `bincode` formats.
+ * @note Ignores the Brillig parts of the bytecode when using `msgpack`.
+ */
+Acir::Program deserialize_program(std::vector<uint8_t> const& buf);
 
 /**
- * @brief Converts from the ACIR-native `WitnessMap` format to Barretenberg's internal `WitnessVector` format.
+ * @brief Deserializes a `WitnessStack` from bytes, trying `msgpack` or `bincode` formats.
+ */
+Witnesses::WitnessStack deserialize_witness_stack(std::vector<uint8_t> const& buf);
+
+/**
+ * @brief Converts from the ACIR-native `WitnessStack` format to Barretenberg's internal `WitnessVector` format.
  *
- * @param buf Serialized representation of a `WitnessMap`.
- * @return A `WitnessVector` equivalent to the passed `WitnessMap`.
+ * @param buf Serialized representation of a `WitnessStack`.
+ * @return A `WitnessVector` equivalent to the last `WitnessMap` in the stack.
  * @note This transformation results in all unassigned witnesses within the `WitnessMap` being assigned the value 0.
  *       Converting the `WitnessVector` back to a `WitnessMap` is unlikely to return the exact same `WitnessMap`.
  */
 WitnessVector witness_buf_to_witness_data(std::vector<uint8_t> const& buf);
 
+AcirFormat circuit_buf_to_acir_format(std::vector<uint8_t> const& buf, uint32_t honk_recursion);
+
 std::vector<AcirFormat> program_buf_to_acir_format(std::vector<uint8_t> const& buf, uint32_t honk_recursion);
 
 WitnessVectorStack witness_buf_to_witness_stack(std::vector<uint8_t> const& buf);
@@ -25,4 +50,4 @@ AcirProgramStack get_acir_program_stack(std::string const& bytecode_path,
                                         std::string const& witness_path,
                                         uint32_t honk_recursion);
 #endif
-} // namespace acir_format
+} // namespace acir_format