Optional fields on pyarrow structs triggers error

[jgalinho]

Hello,

When I try to include an optional field in a pyarrow.struct inside of a Schema and call aggregate_polars_all, the latest version (>=1.6.0) throws the following error:

ArrowInvalid: Mismatching child array lengths

For reference, I include the code used to produce this below. field2 is the list that contains multiple structs in which value2 is an optional value.

pa_struct = {
    "value1": pyarrow.float64(),
    "value2": pyarrow.float64()
}

mongo_projection = {
    "field1": "$field1",
    "field2": "$field2",
}

mongo_schema = {
    "field1": pyarrow.string(),
    "field2": pyarrow.list_(pyarrow.struct(pa_struct))
}

df = col.aggregate_polars_all([
    {"$match": mongo_query},
    {"$project": mongo_projection}
], schema=Schema(mongo_schema))

Here is how the earlier version (<=1.5.2) correctly and conveniently assigns null if the value is not there:

If I try to aggregate it to a pandas DataFrame instead using the aggregate_pandas_all function, it always gives me the following error, regardless of which version I use:

TypeError: Cannot convert numpy.ndarray to numpy.ndarray

Is this intended behavior? Is there possibly a better and intended way to achieve this in the latest version?

Let me know if I can help in providing any more information.

Thanks!

[blink1073]

Hi @jgalinho, thanks for the report! I suspect this will be fixed by #261, but I wasn't able to reproduce your error. Can you please include some sample documents and the value of mongo_query?

[jgalinho]

Took some time to anonymize and strip both the data and the code back to the basics with the bare minimums required to reproduce the issue.

First of all, import this JSON file into a Mongo collection. With the collection loaded with pymongo as col, the following code reproduces the issue:

col.aggregate_polars_all([
    {"$project": {"col": "$list1"}}
], schema=Schema({"col": pyarrow.list_(pyarrow.struct({"field1": pyarrow.float64(),
                                                       "field2": pyarrow.float64()}))}))

Also, I was not able to further replicate the TypeError that was happening with aggregate_pandas_all. But I was able to verify that this same behavior occurs.

Let me know if you were able to replicate the issue and/or if you need any further information. Thank you!

[blink1073]

Thank you very much! I'll take a look and see if we can fix these bugs along with #261 for the next patch release.

[blink1073]

Thanks again! I incorporated your test data and reproduction as tests in the fix PR: #263