Should we register DataFusion UDF at do_connect like in SQLite?

[mesejo]

Currently, each UDF in DataFusion is registered at runtime in the function defined for the operation, for example:

@translate.register(ops.DayOfWeekName)
def extract_down(op, **kw):
    def down(array: pa.Array) -> pa.Array:
        return pc.choose(
            pc.day_of_week(array),
            "Monday",
            "Tuesday",
            "Wednesday",
            "Thursday",
            "Friday",
            "Saturday",
            "Sunday",
        )

    extract_down_udf = df.udf(
        down,
        input_types=[PyArrowType.from_ibis(op.arg.dtype)],
        return_type=PyArrowType.from_ibis(op.dtype),
        volatility="immutable",
        name="extract_seconds_udf",
    )
    arg = translate(op.arg, **kw)
    return extract_down_udf(arg)

So (IIUC) every time the ops.DayOfWeekName is executed the code will try to register the UDF.

So the question is should we instead do it at do_connect like in SQLite? (see here)

[cpcloud]

@mesejo Thanks for the question!

Yeah, I think it could consolidate the code and make it easier for folks to contribute new operation implementations to have a single place to go for referencing and defining UDFs.

Would happily accept PRs!

[mesejo]

Thanks!