ExtensionType API incomplete

[nsslh]

Is the API for supporting MsgPack extensions complete?

(So as not to waste anyone's time, note that this is an adaptation of a question on Discourse.)

I am trying to write the Julia end of an RPC-over-RabbitMQ relationship. The other side (written in Ruby) uses a MsgPack extension for dates. I receive this:

MsgPack.Extension(1, UInt8[0x07, 0xe6, 0x02, 0x1c])

I can make my end send this with pack, but I can’t make my end convert this to a Date with unpack. This is what I have so far, based on the documentation for MsgPack.ExtensionType:

module test_msg_pack_ext

using Dates
using MsgPack
using Test

function pack_date(date::Date)::MsgPack.Extension
        year_u16 = convert(UInt16, year(date))
        year_hi = UInt8(year_u16 >> 8)
        year_lo = UInt8(year_u16 << 8 >> 8)
        return MsgPack.Extension(0x01, UInt8[year_hi, year_lo, month(date), day(date)])
end

function unpack_date(ext::MsgPack.Extension)::Date
        year = ext.data[1] << 8 | ext.data[2]
        return Date(year, ext.data[3], ext.data[4])
end

MsgPack.msgpack_type(::Type{Dates.Date}) = MsgPack.ExtensionType()
MsgPack.to_msgpack(::MsgPack.ExtensionType, date::Dates.Date) = pack_date(date)
MsgPack.from_msgpack(::Type{Dates.Date}, ext::MsgPack.Extension) = unpack_date(ext)

@test pack(MsgPack.Extension(1, UInt8[0x07, 0xe6, 0x02, 0x1c])) == pack(Date(2022, 2, 28))
@test Date(2022, 2, 28) == unpack(UInt8[0xd6, 0x01, 0x07, 0xe6, 0x02, 0x1c])
    
end

# =>
engine/msg_pack_ext.jl: Test Failed at .../msg_pack_ext.jl:25
  Expression: Date(2022, 2, 28) == unpack(UInt8[0xd6, 0x01, 0x07, 0xe6, 0x02, 0x1c])
   Evaluated: Dates.Date("2022-02-28") == MsgPack.Extension(1, UInt8[0x07, 0xe6, 0x02, 0x1c])

That makes sense to me. I haven’t done anything to map extension type 1 to my code. But I can't see anything in the MsgPack code or test suite that lets me do this. The discussion in #7 makes it look like the contributor was satisfied with receiving a MsgPack.Extension and handling it themselves?

[ksixty]

Has any solution for mapping extensions been found? I'm struggling with implementing an ext-based wrapper for 128-bit integers, it seems that there's no way to do that with MsgPack.jl.

[nsslh]

@ksixty I'm not aware of a library-assisted solution. I've had to provide my own custom unpack function. For example:

module MsgPackExt

using Dates
using MsgPack: MsgPack, Extension, ExtensionType

export pack, unpack

extended_unpack(bytes::Vector{UInt8})::Any = _extended_unpack(MsgPack.unpack(bytes))
_extended_unpack(args::Vector{Any})::Vector{Any} = map!(_extended_unpack, args, args)
_extended_unpack(ext::Extension)::Union{Date,Extension} = ext.type == 1 ? unpack_date(ext) : ext
_extended_unpack(arg::Any)::Any = arg

MsgPack.msgpack_type(::Type{Date}) = ExtensionType()
MsgPack.to_msgpack(::ExtensionType, date::Date) = pack_date(date)
# MsgPack.from_msgpack is unused. See https://github.com/JuliaIO/MsgPack.jl/issues/49
MsgPack.from_msgpack(::Type{Date}, ext::Extension) = unpack_date(ext)

function pack_date(date::Date)::Extension
        year_u16 = convert(UInt16, year(date))
        year_hi = UInt8(year_u16 >> 8)
        year_lo = UInt8(year_u16 << 8 >> 8)
        return Extension(0x01, UInt8[year_hi, year_lo, month(date), day(date)])
end

function unpack_date(ext::Extension)::Date
        year = UInt16(ext.data[1]) << 8 | ext.data[2]
        return Date(year, ext.data[3], ext.data[4])
end

pack = MsgPack.pack
unpack = extended_unpack

end

[tscode]

This is not a solution to this issue, but in past projects I have also hit walls regarding serialization in julia and recently decided to roll my own msgpack based serialization package: StructPack.jl.

Using a msgpack extension type to store Int128 (@ksixty) could be done as follows:

using StructPack

I = Int8(5)

# These are maybe not the most efficient ways to do this?
StructPack.destruct(x::Int128, ::ExtensionFormat{I}) = reinterpret(UInt8, [x])
StructPack.construct(::Type{Int128}, data, ::ExtensionFormat{I}) = reinterpret(Int128, data)[1]

val = Int128(10)

bytes = pack(val, ExtensionFormat{I}())
unpack(bytes, Int128, ExtensionFormat{I}())

You can make ExtensionFormat{I} the default format via StructPack.format(::Type{Int128}) = ExtensionFormat{I}(), or use the convenience macro @pack Int128 in ExtensionFormat{I}.

The problem is that this might interact badly with other packages, which might use a different scheme to store Int128. This really annoyed me, since this also affected storing something as simple as Vector{Float64}: should this be a msgpack array of float values, or a msgpack bytes object? It sucks that this is a global choice to make...

For this reason, StructPack includes two ways to more flexibly assign formats. First, if you only want to pack Int128s inside of a specific struct, you can do something like this:

struct A
  a::Int128
  b::Vector{Float64}
end

@pack A in StructFormat [a in ExtensionFormat{I}, b in BinVectorFormat]
# or
StructPack.format(::Type{A}) = StructFormat()
StructPack.fieldformats(::Type{A}) = (ExtensionFormat{I}(), BinVectorFormat())

You can check that this uses the extension format for a and the binary format for b: pack(A(5, rand(2))) |> unpack.

Secondly, you can create your own "context" object and tell StructPack to treat your type a specific way in this context:

struct MyContext <: StructPack.Context end

@pack MyContext Int128 in ExtensionFormat{I}
# or
StructPack.format(::Type{Int128}, ::MyContext) = ExtensionFormat{I}()

bytes = pack(val, MyContext())
unpack(bytes, Int128, MyContext())

# or, if you fancy ScopedValues in julia 1.11+,

using Base.ScopedValues

with(StructPack.context => MyContext()) do
  bytes = pack(val)
  unpack(bytes, Int128)
end

The context is handed further down to all packing / unpacking related calls, so all packing calls internal to pack(..., MyContext()) will know about MyContext.