Add PyarrowScalarUDF
and convert PyScalarUDF
to API recommended upstream
#907
Labels
enhancement
New feature or request
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Up until #906,
PyScalarUDF
has useddatafusion_physical_expr::functions::make_scalar_function
which has been deprecated since v36 and is being removed from the public API in v43.The deprecation warning advised:
datafusion-python
users can not currently implementScalarUDF
in that recommended way.Describe the solution you'd like
PyScalarUDF
to take functions that operate onColumnarValue
, as recommended upstream.PyarrowScalarUDF
that let's python users continue writing UDF's as they do today.Describe alternatives you've considered
We could remove the ability to write pyarrow UDF's, forcing users to migrate as directed in the deprecation warning.
Additional context
It may be useful to review how upstream previously converted functions that use
ArrayRef
toScalarUDF
.https://github.com/apache/datafusion/blob/8e8bb89a285c6fe6e8186378b7798d3d213f08f7/datafusion/physical-expr/src/functions.rs#L79-L122
The text was updated successfully, but these errors were encountered: