Error Instead of Panic On Attempting to Write More Than 32769 Row Groups

[pacman82]

Describe the bug

i16 counting row groups overflows and becomes negative causing panic

To Reproduce

Writing 32769 row groups with the file writer

Expected behavior

Maybe an error indicating that too many batches have been written would be preferable. Alternatively it would be nice if this just worked, yet I could also get behind the thinking that this may be too many row groups for a single file anyway.

Additional context

Occurred in the context of a user running odbc2parquet. His row groups were very small (15 rows) due to an issue with his row sizes, causing him to write lots of row groups into a single file. See: pacman82/odbc2parquet#652

[tustvold]

The i16 is actually limit enforced by the parquet format itself - https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L940

Row groups of this size are such a bad idea, the format actively prevents it 😅

That being said we could make this an error not a panic

[pacman82]

No disagreement here. I am exploring opportunities to change the UX of odbc2parquet in a way to avoid this scenario entirely, but still felt that the panic should be an error.

[alamb]

I agree error vs panic is a much nicer behavior

[etseidl]

Is this fixed by #6378? I can't reproduce with the current head because next_row_group() will no longer return a rowgroup writer with an invalid ordinal.

[alamb]

Is this fixed by #6378? I can't reproduce with the current head because next_row_group() will no longer return a rowgroup writer with an invalid ordinal.

It appears that #6378 (thanks @progval ❤️ ) was released in version 53.1.0
https://github.com/apache/arrow-rs/blob/master/CHANGELOG-old.md#5310-2024-10-02

[alamb]

label_issue.py automatically added labels {'parquet'} from #6629

[pacman82]

Great stuff, thanks!

[alamb]

Kudos to @etseidl -- BTW this should be available on crates.io in about 2 days -- it is included in

• Release arrow-rs / parquet minor version 53.3.0 (November 2024) #6597