Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert BuiltInWindowFunction::{Rank, PercentRank, DenseRank} to a user defined functions #12648

Closed
Tracked by #8709
alamb opened this issue Sep 27, 2024 · 9 comments · Fixed by #12718
Closed
Tracked by #8709
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@alamb
Copy link
Contributor

alamb commented Sep 27, 2024

Is your feature request related to a problem or challenge?

Part of #8709

There is now no difference between "built in" / "prepackaged" scalar and aggregate functions in DataFusion, however there are still some "built in" window functions -- see the current source for BuiltInWindowFunction for the up to date list of what remains

The problem with having two different kinds of window functions is

  1. There are some features that may not be available to User Defined Window Functions that rely on built in
  2. Users can not easily choose which window functions to include or override the behavior if they need something different

Describe the solution you'd like

I would like to remove the "built in" version of this function and convert it to a user defined function

Describe alternatives you've considered

At a high level the process is:

  1. Add a new WindowUDFImpl in the functions-window crate
  2. Port the code from the relevant existing implementation of the the built in functions in datafusion/physical-expr/src/window
  3. Remove the BuiltInWindowFunction variant and then get everything to compile (the compiler will show you where the existing implementations are)

Additional context

Here are some good examples:

@alamb alamb added enhancement New feature or request good first issue Good for newcomers labels Sep 27, 2024
@alamb
Copy link
Contributor Author

alamb commented Sep 27, 2024

I think this is a good first issue for someone skilled in rust that wants to get a handle on how datafusion works. If you hit issues, let us know

@dharanad
Copy link
Contributor

dharanad commented Sep 27, 2024

take
I think i should give up on this, let someone else who is new to this project take this up. Should be a good learning exp for them.

@dharanad dharanad removed their assignment Sep 27, 2024
@hailelagi
Copy link
Contributor

hailelagi commented Sep 27, 2024

take (unless this might be intended for someone else, happy to step back)

@jatin510
Copy link
Contributor

If you don't mind . Can i work on this issue @hailelagi ?

@hailelagi hailelagi removed their assignment Sep 28, 2024
@jatin510
Copy link
Contributor

take

@Omega359
Copy link
Contributor

Convert BuiltInScalarFunction:: -> Convert BuiltInWindowFunction::{NthValue}

@alamb alamb changed the title Convert BuiltInScalarFunction::{Rank, PercentRank, DenseRank} to a user defined functions Convert BuiltInWindowFunction::{Rank, PercentRank, DenseRank} to a user defined functions Sep 29, 2024
@SteNicholas
Copy link
Member

@jatin510, are you working on this ticket? I'm new comer to contribute. Could this ticket assign to me? cc @alamb

@jatin510
Copy link
Contributor

Thanks for reaching out. I’m actually already working on this issue, and it’s my first task in the project as well. I’d like to continue with it. @SteNicholas

@jcsherin
Copy link
Contributor

jcsherin commented Sep 30, 2024

@SteNicholas Will you be interested in picking up ntile window function? It is also pending conversion to a user-defined window function and looks like a good first issue to me.

See https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/window/ntile.rs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants