Is there support for faster serialization format than json for embedding

[honeyspoon]

I'm trying to do embedding calculation in a fast way and I would like to get rid fo the last serialization step to json.
Converting a 1k float vec into a string doesn't seem that great.
Is there support for other binary formats like msgpack or arrow?

[wirthual]

Hi @honeyspoon,

currently there are no other binary formats supported. However to limit the duration of the vector conversion as described we are leveraging the orjson library to reduce the time of serialization since it creates bytes directly.

Did a quick check and it seems that msgpack might be a good alternative to support.

import random
import json
import msgpack
import time
import ormsgpack
import sys
import orjson

random.seed(1)

test = {"embeddings": [random.random() for _ in range(1000)]}

ITER = 10000

print("JSON")
start = time.perf_counter()
for i in range(ITER):
    r1 = json.dumps(test)
    ret = json.loads(r1)
print(sys.getsizeof(r1))
print(time.perf_counter() - start)

print("ORJSON")
start = time.perf_counter()
for i in range(ITER):
    r1 = orjson.dumps(test)
    ret = orjson.loads(r1)
print(sys.getsizeof(r1))
print(time.perf_counter() - start)

print("MSGPACK")
start = time.perf_counter()
for i in range(ITER):
    r1 = msgpack.packb(test)
    ret = msgpack.unpackb(r1)

print(sys.getsizeof(r1))
print(time.perf_counter() - start)

print("ORMSGPACK")
start = time.perf_counter()
for i in range(ITER):
    r1 = ormsgpack.packb(test)
    ret = msgpack.unpackb(r1)
print(sys.getsizeof(r1))
print(time.perf_counter() - start)

Results:

JSON
20302
9.277169871999831
ORJSON
19286
0.7398632289996385
MSGPACK
9048
0.4374393660000351
ORMSGPACK
9048
0.2707707850004226

[honeyspoon]

I decided to just build my own server on top of the engine
and send the raw bytes with numpy

[michaelfeil]

Yes, there is the OpenAI encoding format, that is supported if you are e.g. using oak