Possibly wrong serialization of POCO nullable scalar using `SerializerUtils`

[antomys]

Description

Steps to reproduce

1. Use method :

        SerializerUtils.serializeData(
                out,
                input.getStr2Nullable(),
                ClickHouseColumn.of("str2Nullable", ClickHouseDataType.String, true));

2. Serialize test value, save bytes.
3. Push same JSON into clickhouse using CURL:

curl -s -u default:your_secure_password \
  -X POST "http://localhost:8123/?query=INSERT%20INTO%20test_class%20FORMAT%20JSONEachRow" \
  -H "Content-Type: application/json" \
  --data-binary '
{"str1Nullable":null,"str1NotNullable":"value-1","str2Nullable":"optional-value","str2NotNullable":"value-2"}
'

4. Retrieve pushed entity from ClickHouse using CURL:

  curl -s -u default:your_secure_password \
    "http://localhost:8123/?query=SELECT%20*%20FROM%20test_class%20ORDER%20BY%20str1NotNullable%20LIMIT%201%20OFFSET%20$i%20FORMAT%20RowBinary" \
    --output "record_db.bin"

5. Compare and find out, that serializer in repository does not place NULL byte where should be

Error Log or Exception StackTrace

N/A

Expected Behaviour

Nullable byte is written before scalar value

Code Example

public final class Main {
    private static final String STANDARD_TEST_JSON = "{\"str1Nullable\":null,\"str1NotNullable\":\"value-1\",\"str2Nullable\":\"optional-value\",\"str2NotNullable\":\"value-2\"}";
    private static final String ALL_NON_NULL = "{\"str1Nullable\":\"nullable-present\",\"str1NotNullable\":\"not-null-1\",\"str2Nullable\":\"nullable-present\",\"str2NotNullable\":\"not-null-2\"}";

    public static void main(String[] args) throws IOException {

        var mapper =
                JsonMapper.builder()
                        .configure(MapperFeature.ACCEPT_CASE_INSENSITIVE_PROPERTIES, true) // ← Ignore case
                        .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
                        .build();

        extracted(mapper, STANDARD_TEST_JSON, "STANDARD_TEST_JSON");
        extracted(mapper, ALL_NON_NULL, "ALL_NON_NULL");

        return;
    }

    private static void extracted(JsonMapper mapper, String json, String name) throws IOException {
        var dirEv = mapper.readValue(json, TestClass.class);

        var serializer = new TestClassPOJOConvertor();
        var res = serializer.convert(dirEv);

        Files.write(Path.of(name+".bin"), res);
    }
}
final class TestClass{
    public String str1Nullable;
    public String str1NotNullable;

    public String str2Nullable;
    public String str2NotNullable;


    public String getStr1Nullable() {
        return str1Nullable;
    }

    public String getStr1NotNullable() {
        return str1NotNullable;
    }

    public String getStr2Nullable() {
        return str2Nullable;
    }

    public String getStr2NotNullable() {
        return str2NotNullable;
    }
}

final class TestClassPOJOConvertor extends POJOConvertor<TestClass> {

    @Override
    public void instrument(OutputStream out, TestClass input) throws IOException {
        Serialize.writeString(
                out,
                input.getStr1Nullable(),
                false,
                true,
                ClickHouseDataType.String,
                false,
                "str1Nullable");

        Serialize.writeString(
                out,
                input.getStr1NotNullable(),
                false,
                false,
                ClickHouseDataType.String,
                false,
                "str1NotNullable");

        SerializerUtils.serializeData(
                out,
                input.getStr2Nullable(),
                ClickHouseColumn.of("str2Nullable", ClickHouseDataType.String, true));

        SerializerUtils.serializeData(
                out,
                input.getStr2NotNullable(),
                ClickHouseColumn.of("str2NotNullable", ClickHouseDataType.String, false));
    }
}

Configuration

Client Configuration

No client is used.

Environment

• Cloud
• Connector version: flink-connector-clickhouse-1.17:0.1.1
• Language version: JAVA 17
• OS: MacOS

ClickHouse Server

• ClickHouse Server version:
• ClickHouse Server non-default settings, if any:
• CREATE TABLE statements for tables involved:

        CREATE TABLE test_class (
                `str1Nullable` Nullable(String),
                `str1NotNullable` String,

                `str2Nullable` Nullable(String),
                `str2NotNullable` String
        )
        ENGINE = MergeTree
        ORDER BY str1NotNullable;

• Sample data for all these tables, use clickhouse-obfuscator if necessary

{"str1Nullable":null,"str1NotNullable":"value-1","str2Nullable":"optional-value","str2NotNullable":"value-2"}

{"str1Nullable":"nullable-present","str1NotNullable":"not-null-1","str2Nullable":"nullable-present","str2NotNullable":"not-null-2"}

[antomys]

Inside method

    public static void writeString(OutputStream out, String value, boolean defaultsSupport, boolean isNullable, ClickHouseDataType dataType, boolean hasDefault, String column) throws IOException {

there is

writeValuePreamble(out, defaultsSupport, isNullable, dataType, hasDefault, column, value)

which, i assume, places nullable byte for the ClickHouse.

But the following methods:

    public static void writeArray(OutputStream out, Object value, ClickHouseColumn column) throws IOException {
        SerializerUtils.serializeArrayData(out, value, column);
    }

    public static void writeMap(OutputStream out, Object value, ClickHouseColumn column) throws IOException {
        SerializerUtils.serializeData(out, value, column);
    }

    public static void writeTuple(OutputStream out, Object value, ClickHouseColumn column) throws IOException {
        SerializerUtils.serializeData(out, value, column);
    }

does not have it, which leads to insert issue. Specifically Tuple, which can be:
Tuple(Nullable<String>) and contain nullable entities

[mzitnik]

@antomys, Thanks, will look into it.

[mzitnik]

@antomys, it's a bug in our Java client, we will address it in our next client version