add support for optional fields in records (#6669)

This commit adds support for optional fields in records making
fusing of varied JSON data that arises from common sum types much
more uniform.  The SUP format is updated with a ? following field
names to represent optional fields and _ to represent a field
value that is not present.

While the SuperSQL parser can parse SUP literals with optional
fields, record expressions do not yet support optional fields
as this support will come in a subequent PR.

The vector representation encodes optional field columns by
run-length encoding their presence and generates a slot map
on demand when serializing to BSUP or when dereferencing a field
to in turn generate a vector.Dynamic mixed with Missing error values.

This commit is not backward compatible with previous BSUP versions
and we will update the BSUP version nunber in a subsequent PR prior
to the RINCON release.

Author: mccanne

api/queryio/jsup_test.go

@@ -16,7 +16,7 @@ func TestJSUPWriter(t *testing.T) {
 	const record = `{x:1}`
 	const expected = `
 {"type":"QueryChannelSet","value":{"channel":"main"}}
-{"type":{"kind":"record","id":30,"fields":[{"name":"x","type":{"kind":"primitive","name":"int64"}}]},"value":["1"]}
+{"type":{"kind":"record","id":30,"fields":[{"name":"x","type":{"kind":"primitive","name":"int64"},"opt":false}]},"value":["1"]}
 {"type":"QueryChannelEnd","value":{"channel":"main"}}
 {"type":"QueryError","value":{"error":"test.err"}}
 `

book/src/formats/bsup.md

@@ -186,17 +186,18 @@ Any references to a type ID in the body of a typedef are encoded as a `uvarint`.
 A record typedef creates a new type ID equal to the next stream type ID
 with the following structure:
 ```
---------------------------------------------------------
-|0x00|<nfields>|<name1><type-id-1><name2><type-id-2>...|
---------------------------------------------------------
+----------------------------------------------------------------------
+|0x00|<nfields>|<name1><opt-1><type-id-1><name2><opt-2><type-id-2>...|
+----------------------------------------------------------------------
 ```
 Record types consist of an ordered set of fields where each field consists of
 a name and its type.  Unlike JSON, the ordering of the fields is significant
 and must be preserved through any APIs that consume, process, and emit BSUP records.
 
 A record type is encoded as a count of fields, i.e., `<nfields>` from above,
 followed by the field definitions,
-where a field definition is a field name followed by a type ID, i.e.,
+where a field definition is a field name, followed by its optionality coded
+as a single byte, followed by a type ID, i.e.,
 `<name1>` followed by `<type-id-1>` etc. as indicated above.
 
 The field names in a record must be unique.
@@ -215,6 +216,9 @@ string data.
 > even if the field names available to the dot operator are restricted
 > by language syntax for identifiers.
 
+The optionality byte indicates "optional" for the value 1 and mandatory for
+the value 0.
+
 The type ID follows the field name and is encoded as a `uvarint`.
 
 #### 2.1.2 Array Typedef
@@ -336,14 +340,10 @@ whereby the inner loop need not consult and interpret the type ID of each elemen
 
 #### 2.2.1 Tag-Encoding of Values
 
-Each value is prefixed with a "tag" that defines:
-* whether it is the null value, and
-* its encoded length in bytes.
+Each value is prefixed with a length "tag" indicating its encoded length in bytes.
 
-The tag is 0 for the null value and `length+1` for non-null values where
-`length` is the encoded length of the value.  Note that this encoding
-differentiates between a null value and a zero-length value.  Many data types
-have a meaningful interpretation of a zero-length value, for example, an
+Zero-length values are possible as many data types
+have a meaningful empty interpretation, for example, an
 empty array, the empty record, etc.
 
 The tag itself is encoded as a `uvarint`.
@@ -368,18 +368,19 @@ tend to be zero-filled for small integers.
 
 #### 2.2.3 Tag-Encoded Body of Complex Values
 
-The body of a length-N container comprises zero or more tag-encoded values,
+The body of a length-N container for a complex value
+comprises zero or more tag-encoded values,
 where the values are encoded as follows:
 
-| Type     |          Value                          |
-|----------|-----------------------------------------|
-| `array`  | concatenation of elements               |
-| `set`    | normalized concatenation of elements    |
-| `record` | concatenation of elements               |
-| `map`    | concatenation of key and value elements |
-| `union`  | concatenation of tag and value     |
-| `enum`   | position of enum element                |
-| `error`  | wrapped element                         |
+| Type     |          Value                            |
+|----------|-------------------------------------------|
+| `array`  | concatenation of elements                 |
+| `set`    | normalized concatenation of elements      |
+| `record` | option bits and concatenation of elements |
+| `map`    | concatenation of key and value elements   |
+| `union`  | concatenation of tag and value            |
+| `enum`   | position of enum element                  |
+| `error`  | wrapped element                           |
 
 Since N, the byte length of any of these container values, is known,
 there is no need to encode a count of the
@@ -390,6 +391,17 @@ For sets, the concatenation of elements must be normalized so that the
 sequence of bytes encoding each element's tag-counted value is
 lexicographically greater than that of the preceding element.
 
+For records, when its type has optional fields,
+a bit vector of length NF is encoded as a tag-encoded value of
+floor((NF+7)/8) bytes to indicate the omission of an optional value
+where NF is the number of optional fields in the record.
+The field order of optional fields determines their position
+in the bit vector with bit numbers 0-7 (least significant to most significant)
+in the first byte, number 8-15 in the second byte, and so forth.
+Following the option bits is a concatenation
+of elements comprising the mandatory values and the optional values that are present
+all in field order.
+
 A union value is encoded as a container with two elements. The first
 element, called the tag, is the `uvarint` encoding of the
 positional index determining the type of the value in reference to the
@@ -546,11 +558,12 @@ complex type it represents as described below.
 
 A record type value has the form:
 ```
---------------------------------------------------
-|30|<nfields>|<name1><typeval><name2><typeval>...|
---------------------------------------------------
+---------------------------------------------------------
+|30|<nfields>|<opts>|<name1><typeval><name2><typeval>...|
+---------------------------------------------------------
 ```
 where `<nfields>` is the number of fields in the record encoded as a `uvarint`,
+`<opts>` is a bit vector of length `<nfields>` indicating which fields are optional,
 `<name1>` etc. are the field names encoded as in the
 record typedef, and each `<typeval>` is a recursive encoding of a type value.
 

book/src/formats/jsup.md

@@ -298,7 +298,8 @@ super -f jsup input.sup | jq .
         "type": {
           "kind": "primitive",
           "name": "string"
-        }
+        },
+        "opt": false
       },
       {
         "name": "r",
@@ -311,17 +312,20 @@ super -f jsup input.sup | jq .
               "type": {
                 "kind": "primitive",
                 "name": "int64"
-              }
+              },
+              "opt": false
             },
             {
               "name": "b",
               "type": {
                 "kind": "primitive",
                 "name": "int64"
-              }
+              },
+              "opt": false
             }
           ]
-        }
+        },
+        "opt": false
       }
     ]
   },
@@ -356,7 +360,8 @@ super -f jsup input.sup | jq .
         "type": {
           "kind": "primitive",
           "name": "string"
-        }
+        },
+        "opt": false
       },
       {
         "name": "r",
@@ -373,10 +378,12 @@ super -f jsup input.sup | jq .
                   "kind": "primitive",
                   "name": "int64"
                 }
-              }
+              },
+              "opt": false
             }
           ]
-        }
+        },
+        "opt": false
       }
     ]
   },
@@ -401,7 +408,8 @@ super -f jsup input.sup | jq .
         "type": {
           "kind": "primitive",
           "name": "string"
-        }
+        },
+        "opt": false
       },
       {
         "name": "r",
@@ -430,13 +438,16 @@ super -f jsup input.sup | jq .
                           "name": "string"
                         }
                       ]
-                    }
+                    },
+                    "opt": false
                   }
                 ]
-              }
+              },
+              "opt": false
             }
           ]
-        }
+        },
+        "opt": false
       }
     ]
   },

book/src/formats/model.md

@@ -86,20 +86,19 @@ A field name is any UTF-8 string.
 
 A field value is a value of any type.
 
-In contrast to many schema-oriented data formats, the super data model has no way to specify
-a field as "optional" since any field value can be a null value.
-
-If an instance of a record value omits a value
-by dropping the field altogether rather than using a null, then that record
-value corresponds to a different record type that elides the field in question.
+A field is either mandatory or optional as indicated by its optionality.
+The optionalities of fields in two records must be the same for those
+record types to be equivlanet, e.g., type `{a:string,b?:string}` is distinct
+from `{a:string,b:string}`.
 
 A record type is uniquely defined by its ordered list of field-type pairs.
 
 The type order of two records is as follows:
 * Record with fewer columns than other is ordered before the other.
 * Records with the same number of columns are ordered as follows according to:
      * the lexicographic order of the field names from left to right,
-     * or if all the field names are the same, the type order of the field types from left to right.
+     * secondarily the optionality of a field name for two names that are the same,
+     * or if all the field names and optionalities are the same, the type order of the field types from left to right.
 
 ### 2.2 Array
 

book/src/super-sql/operators/fuse.md

@@ -36,8 +36,8 @@ fuse
 {a:1}
 {b:2}
 # expected output
-{a:1::(int64|null),b:null::(int64|null)}
-{a:null::(int64|null),b:2::(int64|null)}
+{a?:1,b?:_::int64}
+{a?:_::int64,b?:2}
 ```
 
 ---
@@ -64,6 +64,6 @@ fuse
 {a:[1,2]}
 {a:["foo","bar"],b:10.0.0.1}
 # expected output
-{a:[1,2]::[int64|string],b:null::(ip|null)}
-{a:["foo","bar"]::[int64|string],b:10.0.0.1::(ip|null)}
+{a:[1,2]::[int64|string],b?:_::ip}
+{a:["foo","bar"]::[int64|string],b?:10.0.0.1}
 ```

book/src/super-sql/sql/set-ops.md

@@ -183,10 +183,10 @@ fork
 # input
 
 # expected output
-{x:1::(int64|null),y:2::(int64|null),z:null::(int64|null)}
-{x:3::(int64|null),y:4::(int64|null),z:null::(int64|null)}
-{x:5::(int64|null),y:6::(int64|null),z:null::(int64|null)}
-{x:null::(int64|null),y:null::(int64|null),z:2::(int64|null)}
-{x:null::(int64|null),y:null::(int64|null),z:3::(int64|null)}
+{x?:1,y?:2,z?:_::int64}
+{x?:3,y?:4,z?:_::int64}
+{x?:5,y?:6,z?:_::int64}
+{x?:_::int64,y?:_::int64,z?:2}
+{x?:_::int64,y?:_::int64,z?:3}
 ```
 ---

book/src/tutorials/prs.bsup

@@ -476,9 +476,9 @@ fuse
 {y:"foo"}
 {x:2,y:"bar"}
 # expected output
-{x:1::(int64|null),y:null::(string|null)}
-{x:null::(int64|null),y:"foo"::(string|null)}
-{x:2::(int64|null),y:"bar"::(string|null)}
+{x?:1,y?:_::string}
+{x?:_::int64,y?:"foo"}
+{x?:2,y?:"bar"}
 ```
 
 Whereas a type union for field `x` is produced in the following:
@@ -511,7 +511,7 @@ fuse(this)
 {x:"foo",y:"foo"}
 {x:2,y:"bar"}
 # expected output
-<{x:int64|string,y:string|null}>
+<{x:int64|string,y?:string}>
 ```
 
 Since the `fuse` here is an aggregate function, it can also be used with

book/src/tutorials/shaping.md

@@ -14,5 +14,5 @@ script: |
 outputs:
   - name: stdout
     data: |
-      {min:0,max:150,count:102::uint64,size:600}
-      {min:200,max:250,count:51::uint64,size:241}
+      {min:0,max:150,count:102::uint64,size:602}
+      {min:200,max:250,count:51::uint64,size:243}

cmd/super/db/manage/ztests/compact-size.yaml

@@ -12,4 +12,4 @@ script: |
 outputs:
   - name: stdout
     data: |
-      {min:1,max:200,count:2000::uint64,size:1035}
+      {min:1,max:200,count:2000::uint64,size:1036}