Update on "[executorch][runtime] Introduce PteDataMap for weight sharing"

PteDataMap is the NamedDataMap that will live in the runtime. It is used to give delegates access to opaque named data stored in the PTE file. Open to alternative naming suggestions, maybe 'PTEDataMap' or 'ProgramDataMap'?

**Usage**
The PteDataMap is owned by the program, and instantiated at program load time if named_data exists in the PTE file. We introduce usage of 'std::optional' here. I think we can also use executorch::aten::optional to avoid adding standard lib ?

When initializing delegates, the PteDataMap is given to delegate_init. Delegates can retrieve opaque delegate data by key using 'get_data'. This gives them a FreeableBuffer that they can free later.

**Testing**
This test uses the C++ flatbuffer API to build a fake program containing named data. We also creates a temp file with sample data that the data loader can wrap around.

TODO: e2e test once delegate aot is ready and we can generate a file with named data.

**Note**
As the PteDataMap wraps around flatbuffer constructs, the Program must outlive the PteDataMap.

PteDataMap does not implement
- get_metadata; currently, all data stored is opaque. Later, we can implement get_metadata if a backend stores plain tensor data.
- load_into; this is mostly used for the training case, and isn't used by delegates, at least not at the moment

Differential Revision: [D70213646](https://our.internmc.facebook.com/intern/diff/D70213646/)

[ghstack-poisoned]

Author: lucylq

runtime/executor/program.h

@@ -13,7 +13,6 @@
 
 #include <cinttypes>
 #include <cstdint>
-#include <optional>
 
 #include <executorch/runtime/core/data_loader.h>
 #include <executorch/runtime/core/error.h>

runtime/executor/pte_data_map.h

@@ -19,25 +19,30 @@ struct DataSegment;
 } // namespace executorch_flatbuffer
 
 namespace flatbuffers {
-// TODO(T216992074): update internal flatbuffers (v1.12) to match OSS (v24.3.5).
 template <typename T>
 struct Offset;
-#if EXECUTORCH_INTERNAL == 1
+} // namespace flatbuffers
+
+#if EXECUTORCH_INTERNAL_FLATBUFFERS == 1
+// TODO(T216992074): update internal flatbuffers (v1.12) to match OSS (v24.3.5).
+namespace flatbuffers {
 template <typename T>
 class Vector;
 using FlatbufferNamedData =
     flatbuffers::Vector<flatbuffers::Offset<executorch_flatbuffer::NamedData>>;
 using FlatbufferDataSegment = flatbuffers::Vector<
     flatbuffers::Offset<executorch_flatbuffer::DataSegment>>;
+} // namespace flatbuffers
 #else
+namespace flatbuffers {
 template <typename T, typename SizeT>
 class Vector;
 using FlatbufferNamedData = flatbuffers::
     Vector<flatbuffers::Offset<executorch_flatbuffer::NamedData>, uint32_t>;
 using FlatbufferDataSegment = flatbuffers::
     Vector<flatbuffers::Offset<executorch_flatbuffer::DataSegment>, uint32_t>;
-#endif
 } // namespace flatbuffers
+#endif
 
 namespace executorch {
 namespace runtime {

runtime/executor/targets.bzl

@@ -59,7 +59,7 @@ def define_common_targets():
             "//executorch/runtime/core:named_data_map",
             "//executorch/schema:program",
         ],
-        exported_preprocessor_flags = [] if runtime.is_oss else ["-DEXECUTORCH_INTERNAL=1"],
+        exported_preprocessor_flags = [] if runtime.is_oss else ["-DEXECUTORCH_INTERNAL_FLATBUFFERS=1"],
     )
 
     for aten_mode in get_aten_mode_options():

runtime/executor/test/pte_data_map_test.cpp

@@ -55,8 +55,10 @@ class PteDataMapTest : public ::testing::Test {
     // Create segments.
     const flatbuffers::Offset<executorch_flatbuffer::DataSegment>
         segment_arr[2] = {
+            // NOLINT facebook-hte-BadArgumentComment
             executorch_flatbuffer::CreateDataSegment(
                 builder_, /*offset=*/0, /*size=*/kSegmentSizes[0]),
+            // NOLINT facebook-hte-BadArgumentComment
             executorch_flatbuffer::CreateDataSegment(
                 builder_,
                 /*offset=*/kSegmentAlignment * 2,