generated from amazon-archives/__template_Custom
-
Notifications
You must be signed in to change notification settings - Fork 286
Closed
Description
Describe the bug
Data Prepper RDS source does not handle MySQL DECIMAL data types with precision ≥19. When RDS exports snapshots to S3, values ≥2^63 are encoded as FIXED_LEN_BYTE_ARRAY in Parquet format (per AWS documentation). The NumericTypeHandler does not support this encoding, causing pipeline failures.
To Reproduce
- Create MySQL table with DECIMAL(38,0) column containing values ≥2^63
- Export RDS snapshot to S3
- Configure Data Prepper pipeline with RDS source pointing to the export
- Pipeline fails with error:
java.lang.IllegalArgumentException: Unsupported value type. The value is of type: class java.util.ArrayList
at org.opensearch.dataprepper.plugins.source.rds.datatype.mysql.handler.NumericTypeHandler.handleNumericType
Expected behavior
RDS source should handle all MySQL DECIMAL types regardless of precision, converting FIXED_LEN_BYTE_ARRAY (ArrayList) to BigInteger for values that exceed INT64 range.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Projects
Status
Done