Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: expose to_pandas_kwargs in read_parquet with pyarrow backend #59654

Merged
merged 16 commits into from
Nov 21, 2024

Conversation

kleinhenz
Copy link
Contributor

@kleinhenz kleinhenz commented Aug 29, 2024

@kleinhenz kleinhenz changed the title expose to_pandas_kwargs expose to_pandas_kwargs in read_parquet with pyarrow backend Aug 29, 2024
@kleinhenz
Copy link
Contributor Author

@jorisvandenbossche would you be able to review? Tagging you since you mentioned this in #56842.

Copy link
Contributor

@yuanx749 yuanx749 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be better if also add to_pandas_kwargs to read_parquet explicitly.

pandas/io/parquet.py Outdated Show resolved Hide resolved
pandas/io/parquet.py Outdated Show resolved Hide resolved
@kleinhenz
Copy link
Contributor Author

Would be better if also add to_pandas_kwargs to read_parquet explicitly.

I wasn't sure if I should do this since it is an engine specific option. Happy to add it though.

@gtholpadi
Copy link

Looking forward to upgrading to this fix. Due to this issue, I am currently using pickles, which is not ideal.

Copy link
Contributor

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

@github-actions github-actions bot added the Stale label Oct 31, 2024
@mroeschke
Copy link
Member

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.

Copy link
Member

@jorisvandenbossche jorisvandenbossche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the slow follow-up!

Could you also add a parameter description to the docstring?

@jorisvandenbossche jorisvandenbossche added Enhancement IO Parquet parquet, feather and removed Stale labels Nov 12, 2024
@jorisvandenbossche
Copy link
Member

(I see that in #57044, we had some discussion if we actually want this .. Will move that part of the discussion to the issue)

@kleinhenz
Copy link
Contributor Author

@jorisvandenbossche I think I've addressed your comments and resolved merge conflicts.

@jorisvandenbossche jorisvandenbossche changed the title expose to_pandas_kwargs in read_parquet with pyarrow backend ENH: expose to_pandas_kwargs in read_parquet with pyarrow backend Nov 19, 2024
@jorisvandenbossche jorisvandenbossche added this to the 3.0 milestone Nov 19, 2024
Copy link
Member

@jorisvandenbossche jorisvandenbossche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! The code looks good, just two small comments on the docs

doc/source/whatsnew/v3.0.0.rst Outdated Show resolved Hide resolved
pandas/io/parquet.py Outdated Show resolved Hide resolved
Copy link
Member

@jorisvandenbossche jorisvandenbossche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

pandas/io/parquet.py Outdated Show resolved Hide resolved
Co-authored-by: Joris Van den Bossche <[email protected]>
@jorisvandenbossche jorisvandenbossche merged commit 1c986d6 into pandas-dev:main Nov 21, 2024
51 checks passed
@jorisvandenbossche
Copy link
Member

Thanks @kleinhenz!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: Expose to_pandas_kwargs in read_parquet for pyarrow engine
5 participants