feat: mysql chunking optimisation by saksham-datazip · Pull Request #797 · datazip-inc/olake

saksham-datazip · 2026-01-27T11:40:34Z

Description

This PR improves the MySQL chunking strategy with the primary goal of significantly reducing chunk generation time for large tables during incremental reads.

To achieve this, two mathematical chunking strategies were introduced based on the primary key type, replacing repeated database-based chunk discovery.

Numeric Primary Keys

The numeric range [min, max] is divided using an arithmetic progression to generate evenly spaced chunk boundaries. This allows chunk boundaries to be computed mathematically instead of relying on repeated database lookups, significantly reducing chunking time.

String Primary Keys

String values are mapped into a numeric space using Unicode encoding (big.Int) and then split into balanced ranges. These candidate boundaries are then aligned with actual database values using collation-aware queries to maintain correct ordering.

These strategies substantially reduce the number of database round trips required for chunk discovery, resulting in faster chunk generation and improved performance for large datasets.

As part of this work, several edge cases in chunk boundary calculation were also addressed, particularly around MySQL collation-aware ordering for string primary keys. The implementation aligns generated boundaries with actual database values using collation-aware queries, ensuring correct range generation and preventing missing or overlapping chunks.

Additionally, a small compatibility fix was introduced in refractor.go. Previously, some queries used hardcoded SQL strings, which caused MySQL to return numeric values as uint64. After switching to parameterized queries, the Go MySQL driver began returning these values as []uint8 (byte slices).

To handle this change correctly, an additional []uint8 case was added in ReformatInt64 so that these values are properly parsed and converted to int64. This ensures consistent behavior regardless of how the query result is returned by the driver.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Tested MySQL chunking with INT32 primary keys
Tested MySQL chunking with INT64 primary keys
Tested MySQL chunking with FLOAT / DOUBLE primary keys
Verified no data loss or overlap across chunk boundaries
Tested on different kind of string pk for full refresh and cdc
Confirmed performance improvement on large datasets

Performance Stats (Different PK Types)

The following stats.json outputs were collected from runs on different MySQL tables, each containing 10M records, using different primary key types.

🔢 Table with `INT32` Primary Key

Seconds Elapsed: 184.00
Speed: 54,347.30 rps
Memory: 96 MB

🔣 Table with `FLOAT64` Primary Key

Seconds Elapsed: 54.00
Speed: 185,179.58 rps
Memory: 36 MB

Screenshots or Recordings

https://datazip.atlassian.net/wiki/x/AYCVDg

Documentation

Documentation Link: [link to README, olake.io/docs, or olake-docs]
N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

N/A

drivers/mysql/internal/backfill.go

vaibhav-datazip · 2026-01-28T10:57:29Z

drivers/mysql/internal/backfill.go

+		if chunk.Min == nil && chunk.Max == nil && filter == "" {
+			stmt = fmt.Sprintf(
+				"SELECT * FROM `%s`.`%s`",
+				stream.Namespace(),
+				stream.Name(),
+			)


why are we falling back to single chunk , shouldn't this fall back to older way ?

this is added because after where clause all conditions are empty than this statement

select * from stream.Namespace().stream.Name() where

will throw error.

drivers/mysql/internal/backfill.go

vaibhav-datazip · 2026-01-28T11:26:31Z

drivers/mysql/internal/backfill.go

+		err = splitViaPrimaryKey(
+			ctx,
+			m,
+			stream,
+			chunks,
+			chunkColumn,
+			chunkSize,
+			minVal,
+			maxVal,
+			pkColumns,
+		)


don't split it into various lines, call in single line

vaibhav-datazip · 2026-01-28T11:29:42Z

drivers/mysql/internal/backfill.go

+func shouldUseEvenDistribution(
+	minVal any,
+	maxVal any,
+	approxRowCount int64,
+	chunkSize int64,
+) (bool, int64) {


similarly here, figure out other places as well where you have done this and correct it

drivers/mysql/internal/backfill.go

vaibhav-datazip · 2026-01-28T11:46:05Z

drivers/mysql/internal/backfill.go

+func splitEvenlyForInt(minVal, maxVal any, chunks *types.Set[types.Chunk], step int64) {
+	start, _ := typeutils.ReformatFloat64(minVal)
+	end, _ := typeutils.ReformatFloat64(maxVal)
+	logger.Infof("splitEvenlyForInt start=%v end=%v step=%d", start, end, step)


Remove this log

vaibhav-datazip · 2026-01-28T11:59:50Z

drivers/mysql/internal/backfill.go

+		if chunk.Min == nil && chunk.Max == nil && filter == "" {
+			stmt = fmt.Sprintf(
+				"SELECT * FROM `%s`.`%s`",
+				stream.Namespace(),
+				stream.Name(),


query building logic is being handled in jdbc.go file

vaibhav-datazip · 2026-01-28T12:03:58Z

drivers/mysql/internal/backfill.go

+	if start+float64(step) > end {
+		chunks.Insert(types.Chunk{
+			Min: nil,
+			Max: nil,
+		})
+		logger.Infof("Generated single chunk: %+v", chunks)
+		return
+	}
+
+	prev := start
+
+	for next := start + float64(step); next <= end; next += float64(step) {
+		chunks.Insert(types.Chunk{
+			Min: utils.ConvertToString(prev),
+			Max: utils.ConvertToString(next),
+		})
+		prev = next
+	}
+
+	chunks.Insert(types.Chunk{
+		Min: utils.ConvertToString(prev),
+		Max: nil,


this functions logic is too much cluttered can be made simpler also most of this code looks similar to older code

saksham-datazip

self review

drivers/mysql/internal/backfill.go

vaibhav-datazip · 2026-02-09T08:11:06Z

drivers/mysql/internal/backfill.go

+		maxValBaseN, err1 := convertStringToIntBaseN(utils.ConvertToString(maxVal))
+		minValBaseN, err2 := convertStringToIntBaseN(utils.ConvertToString(minVal))
+		if err1 != nil || err2 != nil {
+			return


you should at least return the error , otherwise how will someone debug what happened

drivers/mysql/internal/config.go

vaibhav-datazip · 2026-02-09T09:20:10Z

pkg/jdbc/jdbc.go

+func MySQLTableSizeQuery() string {
+	return `
+		SELECT 
+			DATA_LENGTH + INDEX_LENGTH AS table_size


why are we including index length as well here ? why wont the older query MySQLTableRowStatsQuery not do the work ?

that querry is for average row count and average row size while what this function is returning is schema size

vaibhav-datazip · 2026-02-09T09:22:36Z

pkg/jdbc/jdbc.go

+	if condition == "" {
+		condition = utils.Ternary(extraFilter != "", extraFilter, "1 = 1").(string)
+	}


this will create not so good query, instead you could have added the WHERE also here and appended it in last string

also can you give example of case where the condition string will be empty. because AFAIK condition creates query based on chunk , and always there will be a chunk like

from start to null

this is used in other drivers as well so i continued using it for sake of consistency

vaibhav-datazip · 2026-02-09T09:27:38Z

drivers/mysql/internal/backfill.go

+	}
+	sort.Strings(pkColumns)
+
+	if stream.GetStream().SourceDefinedPrimaryKey.Len() > 0 || chunkColumn != "" {


you can directly check the pkcolumns array here , it has all the info

vaibhav-datazip · 2026-02-09T10:46:37Z

drivers/mysql/internal/backfill.go

+				Min: nil,
+				Max: nil,


what does nil to nil chunk mean ? , at other places as well we have created chunk from nil to start and start to nil if there is no further value . similar can be done and if there is no value in table we don't need to proceed till this function

vaibhav-datazip · 2026-02-09T10:50:41Z

drivers/mysql/internal/backfill.go

 	return min, max, err
 }
+
+func shouldUseEvenDistribution(minVal any, maxVal any, approxRowCount int64, chunkSize int64) (bool, int64, float64, float64) {


add a comment for what this function does and what it returns

vaibhav-datazip · 2026-02-09T10:53:17Z

drivers/mysql/internal/backfill.go

+	minFloat, err1 := typeutils.ReformatFloat64(minVal)
+	maxFloat, err2 := typeutils.ReformatFloat64(maxVal)
+	if err1 != nil || err2 != nil {
+		return false, 0, 0, 0


again why are we not sending the error back ?

vaibhav-datazip · 2026-02-09T11:04:23Z

drivers/mysql/internal/backfill.go

+			Min: nil,
+			Max: *convertIntBaseNtoString(prev),
+		})
+		for next := new(big.Int).Add(prev, chunkdiff); next.Cmp(&maxValBaseN) < 0; next.Add(next, chunkdiff) {


you have already subtracted from maxValBaseN earlier, check again this logic might be flawed

vaibhav-datazip · 2026-02-09T11:11:42Z

drivers/mysql/internal/backfill.go

+		})
+	}
+	if len(pkColumns) == 1 && isEvenDistribution {
+		splitEvenlyForInt(minFloat, maxFloat, chunks, float64(step))


it will be better if you use int as step so that there are well defined boundaries

this float is because of float pk, their we will need it

drivers/mysql/internal/backfill.go

vaibhav-datazip · 2026-03-02T12:37:11Z

drivers/mysql/internal/backfill.go

+	// Supported MySQL string-like PK datatypes
+	var stringTypes = map[string]struct{}{
+		"char":    {},
+		"varchar": {},
+	}
+	//defining boolean to check if string is supported or not
+	stringSupportedPk := false
+
+	if len(pkColumns) == 1 {
+		isNumericAndEvenDistributed, chunkStepSize, minFloat, maxFloat, err = IsNumericAndEvenDistributed(minVal, maxVal, approxRowCount, chunkSize)
+		if err != nil {
+			isNumericAndEvenDistributed = false
+			logger.Infof("Stream %s: PK is not numeric or conversion failed, falling back to string splitting: %v", stream.ID(), err)
+		}
+		if !isNumericAndEvenDistributed {
+			var dataType string
+			query := jdbc.MySQLColumnTypeQuery()
+			err = m.client.QueryRowContext(ctx, query, stream.Name(), pkColumns[0]).Scan(&dataType, &dataMaxLength)
+			if err != nil {
+				return nil, fmt.Errorf("failed to fetch Column DataType and max length %s", err)
+			} else if _, ok := stringTypes[dataType]; ok {
+				stringSupportedPk = true
+				logger.Infof("%s is a string type PK",pkColumns[0])
+				if dataMaxLength.Valid {
+					logger.Infof("Data Max Length: %d", dataMaxLength.Int64)
+				}
+			}


this can be refactored something like this

Suggested change

// Supported MySQL string-like PK datatypes

var stringTypes = map[string]struct{}{

"char": {},

"varchar": {},

}

//defining boolean to check if string is supported or not

stringSupportedPk := false

if len(pkColumns) == 1 {

isNumericAndEvenDistributed, chunkStepSize, minFloat, maxFloat, err = IsNumericAndEvenDistributed(minVal, maxVal, approxRowCount, chunkSize)

if err != nil {

isNumericAndEvenDistributed = false

logger.Infof("Stream %s: PK is not numeric or conversion failed, falling back to string splitting: %v", stream.ID(), err)

}

if !isNumericAndEvenDistributed {

var dataType string

query := jdbc.MySQLColumnTypeQuery()

err = m.client.QueryRowContext(ctx, query, stream.Name(), pkColumns[0]).Scan(&dataType, &dataMaxLength)

if err != nil {

return nil, fmt.Errorf("failed to fetch Column DataType and max length %s", err)

} else if _, ok := stringTypes[dataType]; ok {

stringSupportedPk = true

logger.Infof("%s is a string type PK",pkColumns[0])

if dataMaxLength.Valid {

logger.Infof("Data Max Length: %d", dataMaxLength.Int64)

}

}

// defining boolean to check if string is supported or not

stringSupportedPk := false

if len(pkColumns) == 1 {

// 1. Try Numeric Strategy

isNumericAndEvenDistributed, chunkStepSize, minFloat, maxFloat, err = IsNumericAndEvenDistributed(minVal, maxVal, approxRowCount, chunkSize)

if err != nil {

isNumericAndEvenDistributed = false

logger.Infof("Stream %s: PK is not numeric or conversion failed: %v", stream.ID(), err)

}

// 2. If not numeric, check for supported String strategy

if !isNumericAndEvenDistributed {

var dataType string

// Fetch column type

query := jdbc.MySQLColumnTypeQuery()

if err := m.client.QueryRowContext(ctx, query, stream.Name(), pkColumns[0]).Scan(&dataType, &dataMaxLength); err != nil {

return nil, fmt.Errorf("failed to fetch Column DataType and max length: %w", err)

}

// Check if type is supported

switch dataType {

case "char", "varchar":

stringSupportedPk = true

logger.Infof("%s is a string type PK (type: %s)", pkColumns[0], dataType)

if dataMaxLength.Valid {

logger.Infof("Data Max Length: %d", dataMaxLength.Int64)

}

default:

logger.Infof("Unsupported string PK type: %s", dataType)

}

vaibhav-datazip · 2026-03-02T13:28:41Z

drivers/mysql/internal/backfill.go

+		for i := int64(0); i < int64(5); i++ {
+			temporarychunkdiff := new(big.Int).Set(chunkdiff)
+			temporarychunkdiff.Add(temporarychunkdiff, big.NewInt(i))
+			temporarychunkdiff.Div(temporarychunkdiff, big.NewInt(i+1))
+			curr := new(big.Int).Set(&minValBaseN)
+
+			for j := int64(0); j < expectedChunks && curr.Cmp(&maxValBaseN) < 0; j++ {
+				rangeSlice = append(rangeSlice, convertIntUnicodeToString(curr))
+				curr.Add(curr, temporarychunkdiff)
+			}
+
+			rangeSlice = append(rangeSlice, convertIntUnicodeToString(&maxValBaseN))
+			query, args := jdbc.MySQLDistinctValuesWithCollationQuery(rangeSlice, tableCollationType)
+			rows, err := m.client.QueryContext(ctx, query, args...)
+			if err != nil {
+				return fmt.Errorf("failed to run distinct query: %v", err)
+			}


add a comment to explain this loop also , use more explanative variables instead of I and j

vaibhav-datazip · 2026-03-04T04:43:49Z

pkg/jdbc/jdbc.go

+	//
+	// For upper bounds, it creates:
+	//   (c1 < v1) OR (c1 = v1 AND c2 < v2) OR (c1 = v1 AND c2 = v2 AND c3 < v3)
+	buildBound := func(values []string, isLower bool) (string, []any) {


isLowerBound variable name

following the same pattern as used in mssql

vaibhav-datazip · 2026-03-04T05:11:29Z

pkg/jdbc/jdbc.go

+	// Combine with any additional filter if present.
 	if extraFilter != "" && chunkCond != "" {
-		chunkCond = fmt.Sprintf("(%s) AND (%s)", chunkCond, extraFilter)
+		return fmt.Sprintf("(%s) AND (%s)", chunkCond, extraFilter), args


arg isn't modified here so change might not be required

vaibhav-datazip · 2026-03-04T05:21:29Z

pkg/jdbc/jdbc.go

+	return query, args
+}
+
+func MySQLCountGeneratedInRange(values []string, tableCollationType string, minVal, maxVal string) (string, []any) {


add comments explaining what this function does as well as others missing

vaibhav-datazip · 2026-03-04T05:23:05Z

drivers/mysql/internal/backfill.go

+	if len(pkColumns) > 0 {
+		minVal, maxVal, err = m.getTableExtremes(ctx, stream, pkColumns)
+		if err != nil {
+			return nil, fmt.Errorf("Stream %s: Failed to get table extremes: %v", stream.ID(), err)


use %s for error everywhere

vaibhav-datazip · 2026-03-04T05:26:35Z

drivers/mysql/internal/backfill.go

+			if float64(validChunksCount) >= float64(expectedChunks)*constants.MysqlChunkSizeReductionFactor {
+				logger.Infof("Successfully Generated Chunks using splitEvenlyForString Method for stream %s", stream.ID())
+				for i, val := range rangeSlice {
+					logger.Debugf("Boundary[%d] = %q", i, val)


why are we printing all the boundaries ?

constants/constants.go

drivers/mysql/internal/backfill.go

vaibhav-datazip · 2026-03-04T13:43:43Z

drivers/mysql/internal/backfill.go

+			return nil, fmt.Errorf("Stream %s: Failed to get table extremes: %s", stream.ID(), err)
+		}
+	}
+	//defining boolean to check if string is supported or not


remove some unnecessary comments , where it becomes self explanatory,

also what does string is supported mean ? we are checking if the pk is only string right ? name it accordingly

vaibhav-datazip · 2026-03-05T07:06:06Z

drivers/mysql/internal/backfill.go

+				var val string
+				if err := rows.Scan(&val); err != nil {
+					return fmt.Errorf("failed to scan row: %s", err)
+				}
+				rangeSlice = append(rangeSlice, val)
+			}


after this we should also check for rows.Err()

vaibhav-datazip · 2026-03-05T07:18:06Z

pkg/jdbc/jdbc.go

+			SUM(CASE 
+				WHEN val COLLATE %s >= ? AND val COLLATE %s <= ? 
+				THEN 1 ELSE 0 END)


when will this second condition give value != 0 ?

this will be called when chunkdiff is negative ie., max is lesser than min due to mysql chunking

vaibhav-datazip · 2026-03-05T08:14:47Z

pkg/jdbc/jdbc.go

@@ -427,16 +509,86 @@ func MySQLPrimaryKeyQuery() string {
 }

 // MySQLTableRowStatsQuery returns the query to fetch the estimated row count and average row size of a table in MySQL


change the comment as well

vaibhav-datazip · 2026-03-05T08:17:16Z

pkg/jdbc/jdbc.go

-		CEIL(data_length / NULLIF(table_rows, 0)) AS avg_row_bytes
+		CEIL(data_length / NULLIF(table_rows, 0)) AS avg_row_bytes,
+		DATA_LENGTH,
+		TABLE_COLLATION


what if table collaition is NULL

solved now we will be fetching it from column instead

vaibhav-datazip · 2026-03-05T08:20:35Z

drivers/mysql/internal/backfill.go

+	var (
+		approxRowCount		int64
+		avgRowSize  	    any
+		avgSchemaSize  		int64


rename it to tableDataLength as that will make more sense

vaibhav-datazip · 2026-03-05T08:21:10Z

drivers/mysql/internal/backfill.go

+		rangeSlice := []string{}
+		// Try up to 5 times to generate balanced chunks by slightly adjusting the chunk size each iteration.
+		for retryAttempt := int64(0); retryAttempt < int64(5); retryAttempt++ {
+			temporarychunkdiff := new(big.Int).Set(chunkdiff)


rename it to temporaryChunkDiff

vaibhav-datazip · 2026-03-05T08:21:30Z

drivers/mysql/internal/backfill.go

+			expectedChunks = 1
+		}
+
+		chunkdiff := new(big.Int).Sub(&maxValBaseN, &minValBaseN)


this is number of rows per chunk name it accordingly like chunkLength

vaibhav-datazip · 2026-03-05T09:32:50Z

drivers/mysql/internal/backfill.go

+		logger.Infof("Using splitEvenlyForString Method for stream %s", stream.ID())
+		err = splitEvenlyForString(chunks)
+
+	case len(pkColumns) > 1:


This condition should run when primary key is present

vaibhav-datazip · 2026-03-05T09:37:22Z

drivers/mysql/internal/backfill.go

+	if approxRowCount == 0 {
+		return false, 0, 0, 0, nil
+	}


can this even get triggered ? as we do early return incase row count is 0

vaibhav-datazip · 2026-03-05T09:37:53Z

drivers/mysql/internal/backfill.go

+		return false, 0, 0, 0, nil
+	}
+
+	minFloat, err1 := typeutils.ReformatFloat64(minVal)


why err is defined every time why not use one error

vaibhav-datazip · 2026-03-05T09:40:00Z

constants/constants.go

+	DistributionLower = 0.05
+	DistributionUpper = 100.0


how did we choose these distribution factors ?

it is an assumption and airbyte used it as well

vaibhav-datazip · 2026-03-05T09:55:22Z

drivers/mysql/internal/backfill.go

+		if err != nil {
+			logger.Debugf("Stream %s: PK is not numeric or conversion failed, falling back to string splitting: %v", stream.ID(), err)
+		}


the error is getting thrown even when its not evenly distributed

vaibhav-datazip · 2026-03-05T10:04:20Z

drivers/mysql/internal/backfill.go

+		val1 := convertUnicodeStringToInt(maxValPadded)
+		maxValBaseN.Set(&val1)
+		val2 := convertUnicodeStringToInt(minValPadded)
+		minValBaseN.Set(&val2)


why are we setting value in min and maxbaseN when val1 and val2 is already bigint

vaibhav-datazip · 2026-03-05T10:05:47Z

drivers/mysql/internal/backfill.go

+		if expectedChunks <= 0 {
+			expectedChunks = 1
+		}


can use ternary operator here

vaibhav-datazip · 2026-03-05T10:57:19Z

pkg/jdbc/jdbc.go

+	if len(values) == 0 {
+		return "", nil
+	}


values slice will contain at least one value, when will this get triggered ?

vaibhav-datazip · 2026-03-05T10:58:09Z

pkg/jdbc/jdbc.go

+	}
+
+	unionParts := make([]string, 0, len(values))
+	args := make([]any, 0, len(values)+2)


why are you defining it as slice of any because all values inserted in it are of type string

because args is expected to be of type any by function querryContext itself in its definition

vaibhav-datazip · 2026-03-05T12:16:11Z

drivers/mysql/internal/backfill.go

+			query, args := jdbc.MySQLDistinctValuesWithCollationQuery(rangeSlice, tableCollationType)
+			rows, err := m.client.QueryContext(ctx, query, args...)
+			if err != nil {
+				return fmt.Errorf("failed to run distinct query: %s", err)
+			}


add a info here that few chunks can be generated from this which can have 0 values

vaibhav-datazip · 2026-03-06T07:15:11Z

drivers/mysql/internal/backfill.go

+
+	minFloat, err1 := typeutils.ReformatFloat64(minVal)
+	if err1 != nil {
+		return false, 0, 0, 0, fmt.Errorf("failed to parse minVal: %s", err1)


instead of throwing error log that reformat failed trying string strategy

vaibhav-datazip · 2026-03-06T07:16:14Z

drivers/mysql/internal/backfill.go

 }
+
+// checks if the pk column is numeric and evenly distributed
+func IsNumericAndEvenDistributed(minVal any, maxVal any, approxRowCount int64, chunkSize int64) (bool, int64, float64, float64, error) {


will stepsize 0 mean is evenly distributed false , int that case can we eliminate the variable isevenlydistributed

vaibhav-datazip · 2026-03-06T07:23:17Z

drivers/mysql/internal/backfill.go

@@ -59,12 +64,18 @@ func (m *MySQL) ChunkIterator(ctx context.Context, stream types.StreamInterface,
 }



Add a todo here to separate the chunking functions from this one , that will help in creating unit tests for these functions

vaibhav-datazip · 2026-03-06T07:47:07Z

drivers/mysql/internal/backfill.go

+	)
+
+	tableStatsQuery := jdbc.MySQLTableStatsQuery()
+	err := m.client.QueryRowContext(ctx, tableStatsQuery, stream.Name()).Scan(&approxRowCount, &avgRowSize, &avgSchemaSize, &tableCollationType)


Here instead of using table collation can we use column specific collation instead of?

vaibhav-datazip · 2026-03-09T17:54:50Z

utils/typeutils/reformat.go

+	case []uint8:
+		strVal := string(v)
+		intValue, err := strconv.ParseInt(strVal, 10, 64)
+		if err == nil {
+			return intValue, nil
+		}
+		uintValue, err := strconv.ParseUint(strVal, 10, 64)
+		if err != nil {
+			return int64(0), fmt.Errorf("failed to change []byte %v to int64: %v", v, err)
+		}
+		//nolint:gosec // G115: converting []uint8 to int64 is safe and required for backward compatibility
+		return int64(uintValue), nil
 	case bool:


please check how this might affect backward compatibility

vaibhav-datazip · 2026-03-09T18:28:18Z

drivers/mysql/internal/backfill.go

-func IsNumericAndEvenDistributed(minVal any, maxVal any, approxRowCount int64, chunkSize int64) (int64, float64, float64) {
-	minFloat, err := typeutils.ReformatFloat64(minVal)
+func IsNumericAndEvenDistributed(minVal any, maxVal any, approxRowCount int64, chunkSize int64) (int64, int64, int64) {
+	if exceedsInt64Limits(minVal) || exceedsInt64Limits(maxVal) {


why check min exceeds limits, wont just checking max do the job ?

it will be helpful for string related case where minval can be greater than maxval due to string sorting
i.e., 2 is greater than INT_MAX during string sorting

vaibhav-datazip · 2026-03-09T18:29:40Z

drivers/mysql/internal/backfill.go

+		return 0, 0, 0
+	}
+
+	minInt64, err := typeutils.ReformatInt64(minVal)


rename these as minBoundary and maxBoundary

vaibhav-datazip · 2026-03-09T19:50:33Z

drivers/mysql/internal/backfill.go

+
+		chunkDiff := new(big.Int).Sub(&maxEncodedBigIntValue, &minEncodedBigIntValue)
+		chunkDiff.Add(chunkDiff, new(big.Int).Sub(big.NewInt(expectedChunks), big.NewInt(1)))
+		chunkDiff.Div(chunkDiff, big.NewInt(expectedChunks)) //ceil division set up


chunkDiff -> stringChunkStepSize
temporaryChunkDiff -> adjustedStepSize

vaibhav-datazip · 2026-03-09T19:54:33Z

drivers/mysql/internal/backfill.go

+		return exceedsInt64Limits(string(v))
+	case string:
+		if _, err := strconv.ParseInt(v, 10, 64); err == nil {
+			return false


in case the pk is string we will still get

minVal or maxVal exceeds int64 limits

solved by adding improving the log to

Not a numeric column or maxVal exceeds int64 limits

from

maxVal exceeds int64 limits

vaibhav-datazip · 2026-03-09T20:23:52Z

drivers/mysql/internal/backfill.go

+		chunkDiff := new(big.Int).Sub(&maxEncodedBigIntValue, &minEncodedBigIntValue)
+		chunkDiff.Add(chunkDiff, new(big.Int).Sub(big.NewInt(expectedChunks), big.NewInt(1)))
+		chunkDiff.Div(chunkDiff, big.NewInt(expectedChunks)) //ceil division set up
+
+		rangeSlice := []string{}
+		// Try up to 5 times to generate balanced chunks by slightly adjusting the chunk size each iteration.
+		for retryAttempt := int64(0); retryAttempt < int64(5); retryAttempt++ {
+			temporaryChunkDiff := new(big.Int).Set(chunkDiff)


Tested this for a table with primary key as string, and had set the size of parquet as 2kb , still getting chunk diff as a really high number.

Yes, this result is expected. Both the min and max values are padded to the maximum length of the string data type. Because of this padding, the effective range between them becomes large.

When we compute the chunk difference using:

ceil((max - min) / expectedChunks)

where expectedChunks represents the number of rows per chunk, the calculated value is mostly determined by the padded range of max - min.

As a result, this calculation is not significantly influenced by the Parquet file size, and observing a large chunkDiff value in this case is an expected outcome.

…ceberg

saksham-datazip

self review

saksham-datazip · 2026-03-11T10:30:18Z

constants/state_version.go

@@ -28,9 +28,13 @@ package constants
 //
 //   - Version 4: (Current Version) Unsigned int/integer/bigint map to Int64.


remove current version from it

saksham-datazip · 2026-03-11T10:30:52Z

utils/typeutils/reformat.go

change this todo

feat: mysql chunking optimization

83ebf36

saksham-datazip commented Jan 27, 2026

View reviewed changes

drivers/mysql/internal/backfill.go Outdated Show resolved Hide resolved

saksham-datazip added 2 commits January 27, 2026 17:19

mysql optimization comment resolve

f5766f8

Merge branch 'staging' into feat/mysql-chunking-optimization

443cf94

vaibhav-datazip reviewed Jan 28, 2026

View reviewed changes

Merge branch 'staging' into feat/mysql-chunking-optimization

6fc574c

saksham-datazip had a problem deploying to integration_tests February 2, 2026 09:42 — with GitHub Actions Failure

chore: formatting fix

c09aee8

saksham-datazip had a problem deploying to integration_tests February 3, 2026 06:30 — with GitHub Actions Failure

my-sql-chunking-formatting-resolved

53520de

saksham-datazip had a problem deploying to integration_tests February 3, 2026 09:46 — with GitHub Actions Failure

saksham-datazip commented Feb 3, 2026

View reviewed changes

drivers/mysql/internal/backfill.go Outdated Show resolved Hide resolved

drivers/mysql/internal/backfill.go Outdated Show resolved Hide resolved

mysql-chunking-self-reviewed

3b9fbe7

saksham-datazip had a problem deploying to integration_tests February 3, 2026 09:53 — with GitHub Actions Failure

mysql-chunking-optimization-for-string-pk

8e4ba6a

saksham-datazip had a problem deploying to integration_tests February 7, 2026 15:19 — with GitHub Actions Failure

Merge branch 'staging' into feat/mysql-chunking-optimization

1707ae1

saksham-datazip had a problem deploying to integration_tests February 7, 2026 15:28 — with GitHub Actions Failure

Merge branch 'staging' into feat/mysql-chunking-optimization

feca5a0

vaibhav-datazip had a problem deploying to integration_tests February 9, 2026 08:01 — with GitHub Actions Failure

feat: solved lint issue

ccfb371

saksham-datazip temporarily deployed to integration_tests February 9, 2026 08:10 — with GitHub Actions Inactive

vaibhav-datazip reviewed Feb 9, 2026

View reviewed changes

Merge branch 'staging' into feat/mysql-chunking-optimization

fe4b4b2

saksham-datazip had a problem deploying to integration_tests February 10, 2026 09:13 — with GitHub Actions Failure

feat: mysql chunking optimization review resolved

910246a

saksham-datazip had a problem deploying to integration_tests February 10, 2026 09:52 — with GitHub Actions Failure

feat: resolving-lint-extra-spaces

1eacf5a

saksham-datazip had a problem deploying to integration_tests February 10, 2026 09:56 — with GitHub Actions Failure

feat: lint error resolved

964a2ee

saksham-datazip temporarily deployed to integration_tests March 1, 2026 05:45 — with GitHub Actions Inactive

chore: saperated buildChunkConditionMySQL function from mssql

fa24a2c

saksham-datazip requested a deployment to integration_tests March 2, 2026 12:23 — with GitHub Actions Waiting

chore: fixed buildChunkConditionMySQL function for multiple colummns

8411568

saksham-datazip requested a deployment to integration_tests March 3, 2026 13:33 — with GitHub Actions Waiting

vaibhav-datazip reviewed Mar 4, 2026

View reviewed changes

vishalm0509 reviewed Mar 4, 2026

View reviewed changes

drivers/mysql/internal/backfill.go Show resolved Hide resolved

drivers/mysql/internal/backfill.go Outdated Show resolved Hide resolved

drivers/mysql/internal/backfill.go Outdated Show resolved Hide resolved

drivers/mysql/internal/backfill.go Outdated Show resolved Hide resolved

chore: resolved comment for final-testing

64f31c1

saksham-datazip requested a deployment to integration_tests March 4, 2026 09:35 — with GitHub Actions Waiting

chore: resolved lint error

debd4eb

saksham-datazip requested a deployment to integration_tests March 4, 2026 09:41 — with GitHub Actions Waiting

vaibhav-datazip reviewed Mar 5, 2026

View reviewed changes

vaibhav-datazip reviewed Mar 6, 2026

View reviewed changes

saksham-datazip force-pushed the feat/mysql-chunking-optimization branch from debd4eb to 1876fb5 Compare March 7, 2026 15:33

saksham-datazip requested a deployment to integration_tests March 7, 2026 15:33 — with GitHub Actions Waiting

saksham-datazip requested a deployment to integration_tests March 7, 2026 16:31 — with GitHub Actions Waiting

fix: changes pulled from staging

86a2d91

saksham-datazip force-pushed the feat/mysql-chunking-optimization branch from 0f4e0da to 86a2d91 Compare March 7, 2026 19:13

saksham-datazip requested a deployment to integration_tests March 7, 2026 19:13 — with GitHub Actions Waiting

chore: float and uint8 issue resolved

8ead67e

saksham-datazip requested a deployment to integration_tests March 9, 2026 06:34 — with GitHub Actions Waiting

chore: converted float64 to int64

0caf2aa

saksham-datazip requested a deployment to integration_tests March 9, 2026 11:00 — with GitHub Actions Waiting

vaibhav-datazip reviewed Mar 9, 2026

View reviewed changes

chore: added uint8[] block and took datatype for numeric value from i…

8ccfdd6

…ceberg

saksham-datazip requested a deployment to integration_tests March 11, 2026 10:16 — with GitHub Actions Waiting

saksham-datazip commented Mar 11, 2026

View reviewed changes

chore: self reviewed

7754d72

saksham-datazip requested a deployment to integration_tests March 11, 2026 10:32 — with GitHub Actions Waiting

		@@ -427,16 +509,86 @@ func MySQLPrimaryKeyQuery() string {
		}

		// MySQLTableRowStatsQuery returns the query to fetch the estimated row count and average row size of a table in MySQL

		@@ -59,12 +64,18 @@ func (m *MySQL) ChunkIterator(ctx context.Context, stream types.StreamInterface,
		}

		@@ -28,9 +28,13 @@ package constants
		//
		// - Version 4: (Current Version) Unsigned int/integer/bigint map to Int64.

Conversation

saksham-datazip commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Performance Stats (Different PK Types)

🔢 Table with INT32 Primary Key

🔣 Table with FLOAT64 Primary Key

Screenshots or Recordings

Documentation

Related PR's (If Any):

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saksham-datazip left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saksham-datazip commented Jan 27, 2026 •

edited

Loading

🔢 Table with `INT32` Primary Key

🔣 Table with `FLOAT64` Primary Key