Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate with datafusion to support filters pushdown from SQL #160

Open
xushiyan opened this issue Oct 8, 2024 · 4 comments · May be fixed by #203
Open

Integrate with datafusion to support filters pushdown from SQL #160

xushiyan opened this issue Oct 8, 2024 · 4 comments · May be fixed by #203
Assignees
@jonathanc-n
Labels
feature
Milestone
release-0.3.0

Comments

@xushiyan
Copy link
Member

xushiyan commented Oct 8, 2024

Follow-up feature of #47

Integrate with datafusion on the predicate expression parsing and support pushdown predicates from SQL.
@xushiyan xushiyan added this to the release-0.4.0 milestone Oct 8, 2024
@xushiyan xushiyan mentioned this issue Oct 8, 2024
Closed
@jonathanc-n
Copy link
Contributor

May I be assigned this?

@xushiyan
Copy link
Member Author

@jonathanc-n sure. can you first elaborate on the changes you're gonna make? like what structs and APIs you'll introduce and some high level design to show how it works for the requirement?

@jonathanc-n
Copy link
Contributor

Probably will involve these:

  • Convert DataFusion's Expr objects into Hudi filters added in feat: support partition prune api #119 .
  • Implement the supports_filters_pushdown method in the TableProvider.
  • Change scan method to pass translated filters to Hudi's apis

@xushiyan
Copy link
Member Author

@jonathanc-n sounds good

@xushiyan xushiyan modified the milestones: release-0.4.0, release-0.3.0 Nov 27, 2024
@jonathanc-n jonathanc-n linked a pull request Nov 29, 2024 that will close this issue
Open
@xushiyan xushiyan mentioned this issue Nov 30, 2024
Open
@xushiyan xushiyan moved this from Todo to In Progress in hudi-rs roadmap Nov 30, 2024
This was referenced Nov 30, 2024
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jonathanc-n jonathanc-n

Labels
feature
Projects
hudi-rs roadmap
Status: In Progress
Milestone
release-0.3.0
Development

Successfully merging a pull request may close this issue.

feat: support filter pushdown for datafusion jonathanc-n/hudi-rs
2 participants
@xushiyan @jonathanc-n