#392 Support maxFieldSize

Author: mrotteveel

devdoc/jdp/jdp-2025-07-statement-max-field-size.adoc

@@ -5,8 +5,8 @@
 
 == Status
 
-* Draft
-* Proposed for: Jaybird 7
+* Published: 2025-11-13
+* Implemented in: Jaybird 7
 
 == Type
 
@@ -35,36 +35,57 @@ ____
 ==== 8.3.1 Silent Truncation
 
 The `Statement.setMaxFieldSize` method allows a maximum size (in bytes) to be set.
-This limit applies only to the `BINARY`, `VARBINARY`, `LONGVARBINARY`, `CHAR`, `VARCHAR`, `LONGVARCHAR`, `NCHAR`, `NVARCHAR`, and `LONGNVARCHAR` data types.
+This limit applies only to the (JDBC types) `BINARY`, `VARBINARY`, `LONGVARBINARY`, `CHAR`, `VARCHAR`, `LONGVARCHAR`, `NCHAR`, `NVARCHAR`, and `LONGNVARCHAR` data types.
 If a limit has been set using `setMaxFieldSize` and there is an attempt to read data that exceeds the limit, any truncation that occurs as a result of exceeding the set limit will _not_ be reported.
 ____
 
-Firebird doesn't provide a way to restrict the maximum number of bytes returned that will not result in string truncation errors for longer values, or -- for shorter values -- would result in _"`wrong`"_ client-side length derivations of variable length encodings like UTF8 (which would result in excessive truncation).
+Firebird doesn't provide a way to restrict the maximum number of bytes returned that will not result in string truncation errors for longer values, or -- for shorter values -- would result in "`wrong`" client-side length derivations of variable length encodings like UTF8 (which would result in excessive truncation).
 
-Jaybird considers `BLOB SUB_TYPE TEXT` to be `LONGVARCHAR` and `BLOB SUB_TYPE BINARY` to be `LONGVARBINARY` (though all blob subtypes can be treated that way).
+Jaybird considers `BLOB SUB_TYPE TEXT` to be `LONGVARCHAR` and `BLOB SUB_TYPE BINARY` and other Firebird subtypes to be `LONGVARBINARY`.
+Custom subtypes (negative subtypes), are considered `Types.BLOB`
 
 == Decision
 
 Jaybird 7 implements support for `setMaxFieldSize`/`getMaxFieldSize` using client-side truncation.
-This will be implemented for `CHAR`, `VARCHAR`, `BINARY`, and `VARBINARY`.
+This will be implemented for `CHAR`, `VARCHAR`, `BINARY`, `VARBINARY`, `LONGVARCHAR` and `LONGVARBINARY`
 
 To avoid problems with `RDB$DB_KEY` columns, implementation must ignore the limit for those columns, as indicated by `FieldDescriptor.isDbKey()`.
 
-Decision for handling `LONGVARCHAR`/`LONGVARBINARY` is pending further investigation.
-Currently, we're leaning towards _not_ honouring the maximum field size, or only for `ResultSet.getBytes`/`getString`.
+Custom blob types (i.e. with a negative subtype) are not affected, as Jaybird considers those `Types.BLOB`.
 
 == Consequences
 
-Handling the truncation for `CHAR`, `VARCHAR`, `BINARY`, and `VARBINARY` will be delegated to the `FbStatement` implementation.
-The truncation will be done when receiving the data from the server (from the wire, or from fbclient).
-
-`FbStatement` will receive an `int` property `maxFieldSize` (getter/setter pair) with a default implementation that ignores the set value and returns 0.
-The actual implementation will override this, and use the set value when receiving data from the server.
-
-Caveats or possible pitfalls:
-
-* Given the truncation happens at a set number of bytes, values in a multibyte character set (UTF8) might end in the Unicode replacement character due to truncation before the end of the encoded codepoint.
-* The detection of `RDB$DB_KEY` column will also ignore "`normal`" `BINARY` columns called `DB_KEY`.
+For `CHAR`, `VARCHAR`, `BINARY`, and `VARBINARY`, truncation is implemented in the `FbStatement` implementations in the GDS-ng layer.
+The truncation is done when fetching the data from the server (wire protocol), or from fbclient (native/embedded).
+
+`FbStatement` has an `int` property `maxFieldSize` with a default implementation that ignores the set value and returns `0`.
+The actual implementations override this, and use the set value when receiving data from the server or native client.
+
+For `LONGVARCHAR`/`LONGVARBINARY`, this is implemented in the relevant `FBField` implementations and configured from `FBResultSet`.
+
+* For non-cached blobs, the maximum length is applied for `getBytes` and getters relying on `getBytes`, like the getters for string, numeric, Boolean and datetime types, and `getObject` that returns any of those types, but not to getters returning `Blob`, `Clob`, `InputStream`, or `Reader`.
++
+In case of `InputStream` and `Reader`, such behaviour doesn't conform to the JDBC requirements, but this avoids complications in the implementation.
+We may address this in the future (e.g. maybe if we implement support for `Blob.getBinaryStream(long, long)`).
++
+Supporting `Blob` and `Clob` for `LONGVARCHAR` and `LONGVARBINARY` is already a non-standard extension so -- in our opinion -- not subject to these requirements.
+* For cached blobs (as used in holdable or emulated scrollable result sets), the maximum length is also applied to methods returning `Blob`, `Clob`, `InputStream`, or `Reader`.
+
+Caveats:
+
+* Given the truncation happens at a set number of bytes, values in a multibyte character set (UTF8) might end in the Unicode replacement character (U+FFFD, '`�`') due to truncation before the end of the encoded codepoint.
+* Jaybird will accept any non-negative value for `maxFieldSize`.
+The JDBC apidoc recommends using values greater than `256`.
+* A too small `maxFieldSize` may result in "`wrong`" values being returned without error for numeric and Boolean getters;
+for datetime getters it may result in missing precision without errors, or parse errors.
+* Setting the max field size after execute may not be immediately applied to the current result set for `CHAR`, `VARCHAR`, `BINARY`, and `VARBINARY`:
+** For a locally cached result set: never, as the rows were truncated to the `maxFieldSize` on execute.
+** Otherwise, already fetched rows are truncated to the `maxFieldSize` on their fetch, and only rows returned by a subsequent fetch will apply the new limit.
+* For `LONGVARCHAR`/`LONGVARBINARY`, the value will be truncated on access, with the following caveats:
+** For cached blobs, the limit is applied on retrieval, and applies to all methods (as the cached value is truncated).
+** The value of a cached blob (holdable result set or emulated scrollable result set) will be truncated to the initial `maxFieldSize`.
+Subsequent use of a larger `maxFieldSize` will continue to return the shorter value.
+* The detection of `RDB$DB_KEY` columns includes "`normal`" `BINARY`/`CHAR CHARACTER SET OCTETS` columns called `DB_KEY`.
 
 [appendix]
 == License Notice

jaybird-native/src/main/java/org/firebirdsql/gds/ng/jna/JnaStatement.java

@@ -349,16 +349,16 @@ protected RowValue toRowValue(RowDescriptor rowDescriptor, XSQLDA xSqlDa) {
             if (xSqlVar.sqlind.getValue() == XSQLVAR.SQLIND_NULL) {
                 row.setFieldData(idx, null);
             } else {
-                int bufferOffset;
-                int bufferLength;
-
-                if (rowDescriptor.getFieldDescriptor(idx).isVarying()) {
+                FieldDescriptor fieldDescriptor = rowDescriptor.getFieldDescriptor(idx);
+                int bufferOffset = 0;
+                int bufferLength = switch (fieldDescriptor.getType() & ~1) {
+                case ISCConstants.SQL_VARYING -> {
                     bufferOffset = 2;
-                    bufferLength = xSqlVar.sqldata.getShort(0) & 0xffff;
-                } else {
-                    bufferOffset = 0;
-                    bufferLength = xSqlVar.sqllen & 0xffff;
+                    yield limitToMaxFieldSize(xSqlVar.sqldata.getShort(0) & 0xffff, fieldDescriptor);
                 }
+                case ISCConstants.SQL_TEXT -> limitToMaxFieldSize(xSqlVar.sqllen & 0xffff, fieldDescriptor);
+                default -> xSqlVar.sqllen & 0xffff;
+                };
 
                 byte[] data = new byte[bufferLength];
                 xSqlVar.sqldata.read(bufferOffset, data, 0, bufferLength);
@@ -368,6 +368,10 @@ protected RowValue toRowValue(RowDescriptor rowDescriptor, XSQLDA xSqlDa) {
         return row;
     }
 
+    private int limitToMaxFieldSize(int actualSize, FieldDescriptor fieldDescriptor) {
+        return isApplyMaxFieldSize(fieldDescriptor) ? Math.min(actualSize, maxFieldSize()) : actualSize;
+    }
+
     /**
      * {@inheritDoc}
      * <p>

src/docs/asciidoc/release_notes.adoc

@@ -583,6 +583,43 @@ The `schema` can be `null` or empty string for schemaless tables (i.e. Firebird
 *** `schema()` with the schema, or empty string for schemaless (Firebird 5.0 or older) or if the table was not found
 *** `tableReference()` with the fully qualified and quoted table reference (i.e. `[<quoted-schema>.]<quoted-table-name>`)
 
+[#stmt-max-field-size]
+=== Statement maximum field size (`maxFieldSize`) support
+
+Support for truncating values returned by a result set to the `maxFieldSize` property of a `Statement` was implemented.
+This is a required JDBC feature, but in previous Jaybird 6 and older, the statement implementation only recorded the value, and then ignored it.
+
+The `maxFieldSize` value is defined in bytes, and is applied when the value is greater than zero.
+Truncation occurs for `CHAR`, `VARCHAR`, `BINARY`, `VARBINARY`, `BLOB SUB_TYPE TEXT` (JDBC `LONGVARCHAR`), and `BLOB SUB_TYPE BINARY` and other Firebird built-in (positive) blob subtypes (JDBC `LONGVARBINARY`), but not custom (negative) subtypes (Jaybird considers those JDBC `BLOB`, and JDBC specifies that type should not be limited by `maxFieldSize`).
+
+In case of `CHAR`, `VARCHAR`, `BINARY`, and `VARBINARY`, the truncation is performed when fetching a row (in the GDS-ng layer) _if_ it is not an `RDB$DB_KEY` column.
+This is a client-side truncation, and does not reduce the number of bytes transferred, only the number of bytes retained in memory and returned from the result set.
+
+In the case of built-in (non-negative) blob subtypes, the truncation is performed in the result set field implementation in the JDBC layer.
+This is a client-side truncation, but it may reduce the number of bytes transferred _if_ the blob wasn't already transferred as an inline blob.
+
+For these blob subtypes, the truncation will not occur for methods returning a `Blob`, `Clob`, `InputStream` or `Reader` _if_ the result set is not cached locally (cached result sets are used for holdable result sets and emulated scrollable result sets).
+In the case of `InputStream` and `Reader`, this doesn't conform to the JDBC requirements, but avoided complications in the implementation.
+We may address this in the future.
+
+Caveats:
+
+* Given the truncation happens at a set number of bytes, values in a multibyte character set (UTF8) might end in the Unicode replacement character (U+FFFD, '`&#xFFFD;`') due to truncation before the end of the encoded codepoint.
+* Jaybird will accept any non-negative value for `maxFieldSize`.
+The JDBC apidoc recommends using values greater than `256`.
+* A too small `maxFieldSize` may result in "`wrong`" values being returned without error for numeric and Boolean getters;
+for datetime getters it may result in missing precision without errors, or parse errors.
+* Setting the max field size after execute may not be immediately applied to the current result set for `CHAR`, `VARCHAR`, `BINARY`, and `VARBINARY`:
+** For a locally cached result set: never, as the rows were truncated to the `maxFieldSize` on execute.
+** Otherwise, already fetched rows are truncated to the `maxFieldSize` on their fetch, and only rows returned by a subsequent fetch will apply the new limit.
+* For `LONGVARCHAR`/`LONGVARBINARY`, the value will be truncated on access, with the following caveats:
+** For cached blobs, the limit is applied on retrieval, and applies to all methods (as the cached value is truncated).
+** The value of a cached blob (holdable result set or emulated scrollable result set) will be truncated to the initial `maxFieldSize`.
+Subsequent use of a larger `maxFieldSize` will continue to return the shorter value.
+* The detection of `RDB$DB_KEY` columns includes "`normal`" `BINARY`/`CHAR CHARACTER SET OCTETS` columns called `DB_KEY`.
+
+For further details, see https://github.com/FirebirdSQL/jaybird/blob/master/devdoc/jdp/jdp-2025-07-statement-max-field-size.adoc[jdp-2025-07: Statement Max Field Size].
+
 // TODO add major changes
 
 [#other-fixes-and-changes]
@@ -616,7 +653,7 @@ In Jaybird 7, it no longer ignores these parameters when querying a Firebird 6.0
 
 If you currently pass the "`wrong`" value for these methods, especially `""` (empty string, i.e. only return schemaless objects), you may get no or fewer results than expected.
 
-[float]
+[discrete]
 ===== `schema`
 
 The `schema` parameter performs an exact, case-sensitive match on the schema name, unless it's `null` (i.e. don't filter by schema).
@@ -635,7 +672,7 @@ If your code currently passes `""` (empty string), you need to either replace it
 
 (Unsupported metadata methods are not listed.)
 
-[float]
+[discrete]
 ===== `schemaPattern`
 
 The `schemaPattern` parameter performs a case-sensitive `LIKE` match on the schema name, unless it's `null` (i.e. don't filter by schema).
@@ -780,6 +817,7 @@ If you are confronted with such a change, let us know on {firebird-java}[firebir
 * `FbWireOperations`
 ** The `ProcessAttachCallback` parameter of `authReceiveResponse` was removed, as all implementations did nothing, and since protocol 13, it wasn't only called for the attach response
 ** Interface `ProcessAttachCallback` was removed
+* The internal interface `org.firdsql.jdbc.field.BlobListenableField` was renamed to `BlobField` due to increased responsibilities
 
 [#breaking-changes-unlikely]
 === Unlikely breaking changes

src/main/org/firebirdsql/gds/impl/wire/XdrInputStream.java

@@ -87,9 +87,15 @@ public byte[] readBuffer() throws IOException {
         return readBuffer(fixupLength(readInt()));
     }
 
-    private static int fixupLength(int len) {
-        // Older Firebird versions may return a 32-bit value that is a sign-extended 16-bit value.
-        // Firebird does something similar in remote/protocol.cpp (xdr_cstring_with_limit).
+    /**
+     * Fix up buffer length.
+     * <p>
+     * In some cases, older Firebird versions may return a 32-bit buffer length that is a sign-extended 16-bit value.
+     * This method will only return the lower 16-bits of such values. Firebird does something similar in
+     * {@code remote/protocol.cpp} ({@code xdr_cstring_with_limit}).
+     * </p>
+     */
+    public static int fixupLength(int len) {
         return len >>> 16 == 0xFFFF ? len & 0xFFFF : len;
     }
 

src/main/org/firebirdsql/gds/ng/AbstractFbStatement.java

@@ -1,10 +1,11 @@
-// SPDX-FileCopyrightText: Copyright 2013-2024 Mark Rotteveel
+// SPDX-FileCopyrightText: Copyright 2013-2025 Mark Rotteveel
 // SPDX-FileCopyrightText: Copyright 2019 Vasiliy Yashkov
 // SPDX-License-Identifier: LGPL-2.1-or-later
 package org.firebirdsql.gds.ng;
 
 import org.firebirdsql.gds.ISCConstants;
 import org.firebirdsql.gds.JaybirdErrorCodes;
+import org.firebirdsql.gds.ng.fields.FieldDescriptor;
 import org.firebirdsql.gds.ng.fields.RowDescriptor;
 import org.firebirdsql.gds.ng.fields.RowValue;
 import org.firebirdsql.gds.ng.listeners.*;
@@ -56,6 +57,7 @@ public abstract class AbstractFbStatement implements FbStatement {
     @SuppressWarnings("java:S3077")
     private volatile FbTransaction transaction;
     private String cursorName;
+    private int maxFieldSize;
     private long timeout;
 
     private final TransactionListener transactionListener = new TransactionListener() {
@@ -847,6 +849,47 @@ protected final String getCursorName() {
         return cursorName;
     }
 
+    @Override
+    public final void setMaxFieldSize(int max) throws SQLException {
+        try (var ignored = withLock()) {
+            checkStatementValid(StatementState.NEW);
+            if (max < 0) {
+                throw FbExceptionBuilder.forNonTransientException(JaybirdErrorCodes.jb_invalidStringLength)
+                        .messageParameter("max", Integer.MAX_VALUE, max)
+                        .toSQLException();
+            }
+            maxFieldSize = max;
+        }
+    }
+
+    @Override
+    public final int getMaxFieldSize() throws SQLException {
+        try (var ignored = withLock()) {
+            checkStatementValid(StatementState.NEW);
+            return maxFieldSize;
+        }
+    }
+
+    @Override
+    public final int maxFieldSize() {
+        return maxFieldSize;
+    }
+
+    /**
+     * Determines if max field size should be applied for the value of {@code fieldDescriptor}.
+     *
+     * @param fieldDescriptor
+     *         field descriptor
+     * @return {@code true} if max field size should be applied, {@code false} otherwise
+     * @since 7
+     */
+    protected final boolean isApplyMaxFieldSize(FieldDescriptor fieldDescriptor) {
+        final int maxFieldSize = this.maxFieldSize;
+        return maxFieldSize != 0 && maxFieldSize < fieldDescriptor.getLength()
+                && (fieldDescriptor.isVarying()
+                || fieldDescriptor.isFbType(ISCConstants.SQL_TEXT) && !fieldDescriptor.isDbKey());
+    }
+
     /**
      * @return The timeout value, or {@code 0} if the timeout is larger than supported
      * @throws SQLException

src/main/org/firebirdsql/gds/ng/FbStatement.java

@@ -4,6 +4,7 @@
 package org.firebirdsql.gds.ng;
 
 import org.firebirdsql.gds.BatchParameterBuffer;
+import org.firebirdsql.gds.ng.fields.FieldDescriptor;
 import org.firebirdsql.gds.ng.fields.RowDescriptor;
 import org.firebirdsql.gds.ng.fields.RowValue;
 import org.firebirdsql.gds.ng.listeners.ExceptionListenable;
@@ -644,6 +645,67 @@ default BatchParameterBuffer createBatchParameterBuffer() throws SQLException {
         throw new FBDriverNotCapableException("implementation does not support createBatchParameterBuffer");
     }
 
+    /**
+     * Sets the maximum number of bytes that can be returned for a "string" column ({@code CHAR},
+     * {@code VARCHAR}, {@code BINARY} or {@code VARBINARY}).
+     * <p>
+     * This method is used to fulfil the contract of {@link java.sql.Statement#setMaxFieldSize(int)} for
+     * non-{@code BLOB} columns. If this method is called after a fetch, the new limit will only be applied for rows
+     * returned by a subsequent fetch.
+     * </p>
+     * <p>
+     * Firebird doesn't support limiting the column sizes without producing string truncation errors, so this will only
+     * result in client-side truncation.
+     * </p>
+     * <p>
+     * Implementations must ignore this limit for {@code RDB$DB_KEY} columns (see {@link FieldDescriptor#isDbKey()}).
+     * </p>
+     * <p>
+     * The default implementation in this interface does nothing.
+     * </p>
+     *
+     * @param max
+     *         maximum number of bytes, {@code 0} means there is no limit
+     * @throws SQLException
+     *         if a database access error occurs, this method is called on a closed statement or the condition
+     *         {@code max >= 0} is not satisfied
+     * @see #getMaxFieldSize()
+     * @since 7
+     */
+    default void setMaxFieldSize(int max) throws SQLException {
+    }
+
+    /**
+     * Gets the maximum number of bytes that can be returned for a &quot;string&quot; column ({@code CHAR},
+     * {@code VARCHAR}, {@code BINARY} or {@code VARBINARY}).
+     * <p>
+     * The default implementation in this interface returns 0.
+     * </p>
+     *
+     * @return maximum number of bytes, {@code 0} means there is no limit
+     * @throws SQLException
+     *         if a database access error occurs, this method is called on a closed statement
+     * @see #setMaxFieldSize(int)
+     * @since 7
+     */
+    default int getMaxFieldSize() throws SQLException {
+        return 0;
+    }
+
+    /**
+     * Direct access (no locks, no validity checks) to the max field size.
+     * <p>
+     * The default implementation in this interface returns 0.
+     * </p>
+     *
+     * @return max field size
+     * @see #getMaxFieldSize()
+     * @since 7
+     */
+    default int maxFieldSize() {
+        return 0;
+    }
+
     /**
      * Locks the lock with {@link java.util.concurrent.locks.Lock#lock()} (or equivalent).
      * <p>