Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Expose to_pandas_kwargs in read_parquet for pyarrow engine #57044

Closed
wants to merge 1 commit into from

Conversation

FilipRazek
Copy link

@FilipRazek FilipRazek commented Jan 24, 2024

@FilipRazek FilipRazek force-pushed the fix-49236 branch 10 times, most recently from d77f291 to 5f71823 Compare January 24, 2024 14:03
@WillAyd
Copy link
Member

WillAyd commented Jan 25, 2024

Hmm this design feels weird to me. What arguments would realistically be passed through this?

@WillAyd WillAyd added the IO Parquet parquet, feather label Jan 25, 2024
@FilipRazek
Copy link
Author

In the underlying issue, the OP needs to send timestamp_as_object=True to .to_pandas, to control how the data is formatted as a dataframe.
If it is not a real concern, I can close this PR.

@phofl
Copy link
Member

phofl commented Jan 29, 2024

I am +1 on this, the most important is probably types_mapper, which is a real PITA that this is not possible at the moment

@WillAyd
Copy link
Member

WillAyd commented Jan 29, 2024

Something like read_pandas(to_pandas_kwargs=...) feels strange to me from an API perspective, and has the downside of really intertwining the APIs of the libraries. If we later decide to change the allowable to_pandas_kwargs or pyarrow decides to change their API it would be quite messy

Why not just encourage users to call pa.parquet.read_table in such a case directly?

@FilipRazek
Copy link
Author

Would updating the documentation be sufficient in your opinion?

Copy link
Contributor

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

@github-actions github-actions bot added the Stale label Mar 24, 2024
@FilipRazek
Copy link
Author

@WillAyd ?

@WillAyd
Copy link
Member

WillAyd commented Mar 25, 2024

Sure I think documenting this is a reasonable approach

@mroeschke
Copy link
Member

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
IO Parquet parquet, feather Stale
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: Expose to_pandas_kwargs in read_parquet for pyarrow engine
4 participants