All node metadata re-design

[dehann]

To decide

• Does metadata have crud?
• What is metadata structure?
• How is metadata serialized?

Metadata Structure
Look into:

• JSON3.Object ❌ (JSON3.object is not mutable)
  • JSON3.Object fast enough for Compute_Node_ level?
  • Delete smalldata types union
  • Allows better nesting for JSON types
• key-value Dict{Symbol, SmallDataTypes} (existing)
  Reminder SmallDataTypes are basically JSON values excluding null and objects (nesting):

  const SmallDataTypes = Union{
      Int,
      Float64,
      String,
      Bool,
      Vector{Int},
      Vector{Float64},
      Vector{String},
      Vector{Bool},
  }

• Dict{Symbol, String}
• Custom Struct/Dict

Serialization

• JSON string
• Base64 encoded JSON string (current)
• Modelled Structure

TAC:

• The Metadata design is documented in this issue and approved for DFG v1

---

Design Summary (WIP)

SmallData/Metadata does not follow one design on all nodes. We have:

Metadata for Blobentry

• No CRUD, Blobentry is immutable. Should Blobentry.metadata dict have CRUD? #1172
• Julia definition metadata::JSON.JSONText Should Blobentry.metadata dict have CRUD? #1172
• Limit to JSON object only in constructor.

Bloblet for Agent, Graph, Variable, Factor

• CRUD, we support add, delete, merge, get, etc.
• Julia definition:

struct Bloblet
    label::Symbol
    val::String
end

• Serialized the same as other nodes, ie. ::Vector{Bloblet}.
• Example json

[ {"label": "a_int", "val": "1" }, {"label": "a_str", "val": "one" } ]

[Affie]

My vote is to keep the design as is with SmallDataTypes. Its been this way for 4 years without any issues. see #576

Edit: might change my mind pending the CRUD decision.

[Affie]

Partially addressed in #1107

[Affie]

Also considered just using Dict{Symbol, String} or creating a new struct such as

struct MetadataT
    int::Dict{Symbol, Vector{Int}}
    float::Dict{Symbol, Vector{Float64}}
    string::Dict{Symbol, Vector{String}}
    bool::Dict{Symbol, Vector{Bool}}
end