DataFrame.explode returns a PY[object] column, even if coercible to something better

[xcharleslin]

To Reproduce

from daft import DataFrame

df = DataFrame.from_pydict({"a": [[1, 2, 3]]})
df = df.explode(df["a"])
df.show()

+--------------+
|            a |
|   PY[object] |
+==============+
|            1 |
+--------------+
|            2 |
+--------------+
|            3 |
+--------------+

# Same result even if the list elements are explicitly e.g. pa.int8()s
# df = DataFrame.from_pydict({"a": [pa.array(list(range(10)), type=pa.int8())]})

Expected behavior

+--------------+
|            a |
|      INTEGER |
+==============+
|            1 |
+--------------+
|            2 |
+--------------+
|            3 |
+--------------+

[jaychia]

Thanks for raising this!

This is expected behavior since we can't currently infer any return types of operations on Py columns without some form of user input.

For example, if a user's Python list looks like this:

df = DataFrame.from_pydict({
    "a": [
        [1, 2, 3],
        ["foo", "bar", "baz"],
    ],
})

We can only deduce that the type of the "a" column is a list but Daft has no typing information about its contents at static time. Without knowing the contents of the data itself ahead of time, Daft cannot make any decisions about the elements' types at static time.

However, by casting the resulting exploded dataframe, the user is asserting that the data is of a given type and Daft will throw a runtime error if that assertion proves to be false.

We could perhaps include a way to cast the data from the explode method, similar to the return_type= kwarg in .apply to make this more convenient behavior

[jaychia]

Additionally, in the second case (df = DataFrame.from_pydict({"a": [pa.array(list(range(10)), type=pa.int8())]})) Daft will have support for native nested arrow types and explodes of a list<int64> should indeed give the correct type information!

#335 tracks the issue for adding complex arrow types

[jaychia]

Closing and converting issue to a discussion for future reference!