Skip to content

Standardization does not preserve metadata of nested fields #56

@yruslan

Description

@yruslan

Describe the bug

Standardization keeps metadata of root fields, but does not preserve metadata of nested fields.

To Reproduce

Run standardization of a dataframe with a following schema:

{
  "type": "struct",
  "fields": [
    {
      "name": "STRING_FIELD",
      "type": "string",
      "nullable": true,
      "metadata": {
        "maxLength": 8
      }
    },
    {
      "name": "NESTED_STRUCT",
      "type": {
        "type": "struct",
        "fields": [
          {
            "name": "NESTED_STRING",
            "type": "string",
            "nullable": true,
            "metadata": {
              "maxLength": 5
            }
          }
        ]
      }
    }
  ]
}

The result will look like this (nested field's metadata is lost):

{
  "type": "struct",
  "fields": [
    {
      "name": "STRING_FIELD",
      "type": "string",
      "nullable": true,
      "metadata": {
        "maxLength": 8
      }
    },
    {
      "name": "NESTED_STRUCT",
      "type": {
        "type": "struct",
        "fields": [
          {
            "name": "NESTED_STRING",
            "type": "string",
            "nullable": true,
            "metadata": { }
          }
        ]
      }
    }
  ]
}
  • STRING_FIELD has metadata retained.
  • NESTED_STRING has list metadata field "maxLength": 5

Expected behavior

Standardization should preserve all metadata.

Screenshots

--

Desktop (please complete the following information):

--

Additional context

--

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions