Implement new "datatype" for blob handling

[livius2]

Hi

currently we have BLOB field which require additional call to retrive it.
It is so so slow.
Consider implementing different datatype to handle "blob" data. It should be part of record transmission as normal field.
Especially when this data column is readonly, like in belolw query result of LIST function.

Below comparision queries for speed test. Test it from remote connection (e.g. 20-30 ms ping).
I have tested on FB3 and FB4. First query uses blob from LIST function. Second have CAST list to VARCHAR(32000) and i have added additional ~1KB of data to every record, to show that much more data retrived, but still ~60x faster!

query 1. 15089 ms records 540
query 2. 253 ms records 540

query 1.

WITH RECURSIVE CTE AS (

SELECT RF.RDB$RELATION_NAME, RF.RDB$FIELD_NAME, CAST(1 AS INTEGER) AS LEVEL FROM RDB$RELATION_FIELDS RF WHERE RF.RDB$SYSTEM_FLAG = 1
UNION ALL
SELECT C.RDB$RELATION_NAME, C.RDB$FIELD_NAME, C.LEVEL + 1 FROM CTE C WHERE C.LEVEL<10
)

SELECT 
X.RDB$RELATION_NAME, X.LEVEL, LIST(DISTINCT X.RDB$FIELD_NAME) AS SEP
FROM
CTE X
GROUP BY X.RDB$RELATION_NAME, X.LEVEL

query 2.

WITH RECURSIVE CTE AS (

SELECT RF.RDB$RELATION_NAME, RF.RDB$FIELD_NAME, CAST(1 AS INTEGER) AS LEVEL FROM RDB$RELATION_FIELDS RF WHERE RF.RDB$SYSTEM_FLAG = 1
UNION ALL
SELECT C.RDB$RELATION_NAME, C.RDB$FIELD_NAME, C.LEVEL + 1 FROM CTE C WHERE C.LEVEL<10
)

SELECT 
X.RDB$RELATION_NAME, X.LEVEL, CAST(LIST(DISTINCT X.RDB$FIELD_NAME) AS VARCHAR(32000)) || REPLACE('aaaaaaaaaaaaaaaaaaaaaaaaaaa', 'a', uuid_to_char(gen_uuid())) AS SEP
FROM
CTE X
GROUP BY X.RDB$RELATION_NAME, X.LEVEL

If you like more records to test, simply change C.LEVEL<10 to something else like 'C.LEVEL<50`

[mrotteveel]

This could be solved by other means than a new data type, for example by streaming the blobs inline in a result set, or maybe some compound format that contains the blob ID and the first x bytes of the blob (e.g. +/- 32KB or the actual length if it is less).

For example, by introducing a blr_blob3 to replace blr_blob2, and having the content of the message data be <8-bytes blob id><2-bytes length><data> where given the existing limitations of the message, <data> would be max 32KiB - 10 bytes. Or alternatively, if those limits are dropped, we could use <8-bytes blob id><4-bytes length><data> and maybe make it configurable how much blob data is sent in a message.

If a blob id from a blr_blob3 message is opened, it will be positioned at the first byte after the returned data (or after the end of the blob).

[aafemt]

If you know for sure that your data fits 32k - you don't need BLOBs. And if your data don't fit 32k - you don't want to transfer them through a thin wire without explicit request.

[mrotteveel]

If you know for sure that your data fits 32k - you don't need BLOBs. And if your data don't fit 32k - you don't want to transfer them through a thin wire without explicit request.

In a lot of use cases for blobs, the majority of blobs may be relatively small, while there are a few outliers which are larger. So you use blobs to also be able to store the outliers, or because the fact blobs are stored out-of-band has a benefit when querying things that don't involve the blob data. Not to mention that you may need it to avoid the 64KiB row limit.

Returning the first ~32KiB inline will return all data for cases where the blob is actually smallish, and for cases where the blob is larger, you will already have the first 32KiB available for processing, which could very well improve performance even for larger blobs.

[hvlad]

This could be solved by other means than a new data type, for example by streaming the blobs inline in a result set

I also beleive it is a way to go. To make it more flexible I can think about per-statement or per-attachment setting that control if blob data should be streamed inline or not. Also, it could be useful to set the number of bytes to stream inline.

And no, I don't like blr_blob3, I see no need in it.

[mrotteveel]

And no, I don't like blr_blob3, I see no need in it.

Well, I needed something the distinguish from the current behaviour of blr_blob2 which will only report the blob id.

[asfernandes]

Why a setting?

select blob_field from table is an explicit request that one needs a blob, and faster is preferred than slow.

[hvlad]

To let user decide.

[AlexPeshkoff]

On 9/6/23 10:29, Mark Rotteveel wrote: This could be solved by other means than a new data type, for example by streaming the blobs inline in a result set, or maybe some compound format that contains the blob ID and the first x bytes of the blob (e.g. +/- 32KB or the actual length if it is less).

Using new datatype, where first N (not too big value) bytes of BLOB is stored inline, may have one more advantage - such field may be indexed. Something like Paradox used to have such fields.

[hvlad]

select blob_field from table is an explicit request that one needs a blob, and faster is preferred than slow.

Many apps implemented in a following way: it have a grid with many fields and shows a blob(s) content on demand, when user pressed a button. Almost always the resultset behind a grid contains all fields, including a blob ones. There is no need and no way to show all blobs from all records at the same time. If we will always include blob data into resultset, it will slow such apps significantly. More, it will force users to exclude blob fields from main resultset and create additional request just to fetch desired blob - this is not faster than current separate blob open\fetch\close.

[hvlad]

Using new datatype, where first N (not too big value) bytes of BLOB is stored inline

We speak about storage or about network transfers here ?
Blob storage is another theme, I believe.

may have one more advantage - such field may be indexed. Something like Paradox used to have such fields.

Couldn't expression indices serve this goal ?

[asfernandes]

Using new datatype, where first N (not too big value) bytes of BLOB is stored inline, may have one more advantage - such field may be indexed. Something like Paradox used to have such fields.

How could applications degrades to have the data not indexed (or fully indexed) where the field length increases?

[AlexPeshkoff]

On 9/6/23 13:10, Vlad Khorsun wrote: Using new datatype, where first N (not too big value) bytes of BLOB is stored inline We speak about storage or about network transfers here ? Blob storage is another theme, I believe.

Yes, certainly. But for sure related.

may have one more advantage - such field may be indexed. Something like Paradox used to have such fields. Couldn't expression indices serve this goal ?

Am I wrong that to make use of expression index in a plan that expression should be used in WHERE clause?

[hvlad]

Blob storage is another theme, I believe.

Yes, certainly. But for sure related.

I don't think so, could you explain ?

Couldn't expression indices serve this goal ?

Am I wrong that to make use of expression index in a plan that expression should be used in WHERE clause?

You correct, but... I still see no relation of index thing with storage details nor with the subject.

[AlexPeshkoff]

On 9/6/23 16:18, Vlad Khorsun wrote: Blob storage is another theme, I believe. Yes, certainly. But for sure related. I don't think so, could you explain ?

If we introduce new datatype it's quite logocal to transfer it over the wire in same way as it\s stored on disk. Certainly that's not absolute requirement, it does not make thingsidentical- but for sure relationship.

Couldn't expression indices serve this goal ? Am I wrong that to make use of expression index in a plan that expression should be used in WHERE clause? You correct, but... I still see no relation of index thing with storage details nor with the subject.

I wanted to say that in order to use expression index one has to use not obvious tricks in SQL statements. With discussed datatype it can be WHERE fieldOfNewBlobType starting with 'ABC'

[aafemt]

Why invent a new datatype when IStatement::openCursor() has enough room in "flags" parameter for CURSOR_PREFETCH_BLOBS flag?