How to skip some values when decoding, while preserving the contents

[PlainBane]

Hi,
Is there a way to delay / skip decoding of some values. In Go https://github.com/fxamacker/cbor there is cbor.RawMessage type.
Basically, I have object

`{ "key": "message1",  // Need to read this 
   "data": {...},               // Large, nested object, need to skip this
   "meta": {...}               // Some small data might need to be read here 
  }`

I have a proxy which needs to decode a message, get the key, some metadata and encode it again. Message is then routed to the actual receiver and fully decoded.
I looked at the addExtension, but I'm not quite sure how to just skip a value while keeping it intact.

[kriszyp]

It seems like there some potentially interesting things we could do here.

So to review what is possible right now: you can certainly encode a CBOR object to a byte array/buffer and then encode that into a parent object:

import { encode, decode } from 'cbor-x'
let data = {... large nested object...}
let parent = {
  key: "message1",
  data: encode(data),
}
let encoded = encode(parent);
// and then decoding:
parent = decode(encoded) // can execute pretty quickly without having to decode nested data
parent.key -> "message1"
parent.data -> encoded CBOR buffer
// I can add some data while keeping data encoded and intact
parent.meta = {...}
encode(parent)
// and if I want to decode data and access contents
data = decode(parent.data)
data.someProp

However, we can certainly imagine some improvements on this. In this scheme, the nested data is pretty opaque; it is just encoded as a plain embedded byte array/buffer, and it is up to the consumer to know that accessing the structured contents of data requires decoding it again. It may be desirable to "tag" this data in a way that can be automatically identified on decoding, and perhaps accessed through a JS proxy object that decodes on first property access. And yes, we could do this with an extension:

import { addExtension,... } from cbor-x
class Deferred {}
addExtension({
  Class: Deferred,
  tag: 55801,
  decode(buffer) {
    let decoded
    return new Proxy({}, {
      get(target, prop) {
        if (!decoded) {
          Object.assign(target, decode(buffer))
          decoded = true
        }
        return target[prop]
      }
    })
  },
  encode(data, encode) {
    // double encode it
    return encode(encode(data));
  }
})

// and then:
let data = new Deferred()
Object.assign(data, {... large nested object...})
let parent = {
  key: "message1",
  data,
}
let encoded = encode(parent);
...
parent = decode(encoded)
parent.data.someProp // data is decoded when a property is accessed

And additionally you may want to set this up in a way that a data could be reserialized without any extra decoding/encoding effort (which I didn't include in this example or test at all).

Furthermore, is the question of whether it might be helpful for such proxy-style handling of embedded CBOR data to be actually included in cbor-x itself (so you don't have to write the extensions). And this would also raise a question of the most appropriate/interoperable way of encoding such a thing. This could be encoded with a self-describing CBOR tag (tag 55800) within the byte array, where the contents could be auto-detected. Or that tag or another tag could be around the byte array as indicator of the contents without sniffing in the byte array itself.

Anyway, let me know if you have any thoughts on that, it might be interesting to build in some support for this.

[PlainBane]

Thank you for the detailed explanation. My data flow, currently, is Go fxamacker/cbor --> Node.js cbor-x --> Browser cbor-x and back. I had some miss-mach on the Go side (i was using cbor.RawMessage instead of []byte ), which confused me on the Node.js side. Your explanation made it all clear. The Extension part is super clever, didn't even think of that, will come in handy in the browser. As for embedding this in the library. As far as I can tell I'm the first who has asked for this, so it seems to be an edge case. Unless there is more demand, I would leave it as is. No reason to bloat the library (mainly the browser-land). Auto detecting contents sounds nice, but how many will need this. Maybe add a reference to this in README, if someone looks for a similar thing.
Again, many thanks.