-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
Standardization keeps metadata of root fields, but does not preserve metadata of nested fields.
To Reproduce
Run standardization of a dataframe with a following schema:
{
"type": "struct",
"fields": [
{
"name": "STRING_FIELD",
"type": "string",
"nullable": true,
"metadata": {
"maxLength": 8
}
},
{
"name": "NESTED_STRUCT",
"type": {
"type": "struct",
"fields": [
{
"name": "NESTED_STRING",
"type": "string",
"nullable": true,
"metadata": {
"maxLength": 5
}
}
]
}
}
]
}The result will look like this (nested field's metadata is lost):
{
"type": "struct",
"fields": [
{
"name": "STRING_FIELD",
"type": "string",
"nullable": true,
"metadata": {
"maxLength": 8
}
},
{
"name": "NESTED_STRUCT",
"type": {
"type": "struct",
"fields": [
{
"name": "NESTED_STRING",
"type": "string",
"nullable": true,
"metadata": { }
}
]
}
}
]
}STRING_FIELDhas metadata retained.NESTED_STRINGhas list metadata field"maxLength": 5
Expected behavior
Standardization should preserve all metadata.
Screenshots
--
Desktop (please complete the following information):
--
Additional context
--
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working