feature: search cbor for an element at a given (JSON pointer like) path

[elv-gilles]

Is your feature request related to a problem?
In the use case I'm working on, data is stored in a single CBOR structure.

Users provide input data via a REST API (as JSON) which is stored at conventional location(s) in the global structure.
Server processes then need to retrieve (and potentially modify) these data and/or create new one, also added in the structure.
This results in semi-large to large data.

The data flow currently consists in:

• decoding the global structure into a generic map[string]interface{}
• retrieving the sub-tree in the generic structure that corresponds to the specifics of the process via a JSON-pointer-like parameter
• decode the sub-tree into the actually needed go structs via mapstructure (or other means, including re-encoding into json followed by json unmarshaling)

Describe the solution you'd like
It would be more efficient to be able to locate the sub-tree in cbor and decode into structs from there

Solution would be a function Search(keys []string) (offset int, err error) allowing to decode the element at the returned offset.

Describe alternatives you've considered
see description

Additional context
n/a

[elv-gilles]

I implemented this in a branch that is ready here.
Please let me know if you are interested in a PR.

[fxamacker]

Hi @elv-gilles, thanks for opening this issue and providing a link to your implementation!

This feature seems useful.

I implemented this in a branch that is ready here.
Please let me know if you are interested in a PR.

I like your approach of having a separate Searcher that avoids modifying the decoder, and using syntax similar to JSON pointer.

However, there are some security considerations. The implementation converts CBOR map keys to string (if key is not a CBOR text string), and such conversions can create duplicate map keys.

For example, if a CBOR map has two different keys 123 and "123", then converting integer 123 to string "123" would create a duplicate map key.

Some quick thoughts for consideration:

• Searcher probably need to support all map related decoding options (e.g. DecOptions.DupMapKey) so CBOR map rejected in current approach is also rejected in the new approach.
• Maybe limit supported map key types to scalar types.
• Floating point keys might have edge cases for float32 vs float64, NaNs, etc.
• Maybe use keys []any instead of keys []string, so map keys are typed and don't need to be converted to string for comparison.

JSON Pointer [RFC6901] doesn't distinguish map key vs array index in the path. This can create ambiguity in a CBOR implementation during traversal, especially if we support map key of non-string types without conversion. The same path can be applied to different data structures. For example, [0, 0] can be a valid path for nested array in an array, nested map in a map (with integer map keys), etc. To resolve this, maybe include a path type (map key or array index) along with the value. For example, instead of [0, 0], we can specify the type to eliminate ambiguity between array index and map key: [{Type: ArrayIndex, Value: 0}, {Type: MapKey, Value: 0}].

Thoughts?

[elv-gilles]

Hi @fxamacker, thanks for the answer.

I'll look into hardening my implementation with respect to your comments.

On the other hand, as you noticed, the Searcher is entirely implemented via the decoder.
An alternative would consist in exposing a partial view of the decoder API (make it public) and let users of the cbor/v2 library implement features (like search) by themselves. I made a quick trial at this with a ReadSeekDecoder.

What's your opinion ?

[fxamacker]

On the other hand, as you noticed, the Searcher is entirely implemented via the decoder.
An alternative would consist in exposing a partial view of the decoder API (make it public) and let users of the cbor/v2 library implement features (like search) by themselves. I made a quick trial at this with a ReadSeekDecoder.

What's your opinion ?

Hi @elv-gilles, thanks for the reply!

In most cases, my preference is to separate the high-level API from the low-level API. For example, low-level API can be in its own package, so users can choose the set of API that they are most comfortable with.

I already have an issue open for creating streaming API. Once streaming API is created, I think it can be used for features (like search) under the hood, similar to ReadSeekDecoder.

Before creating the new streaming API (issue #666), I need to wrap up some in-progress refactoring (e.g., issue #709) that were postponed due to more time-sensitive projects that required weekends, holidays, etc.

[elv-gilles]

Hi @fxamacker ok thanks, then I'm putting this on hold (and I think on my side this is not urgent).