Describe the bug, including details regarding any error messages, version, and platform.

When you call to_pandas, Arrow converts ints to doubles This leads to precision issues (e.g., some ints can't be represented with doubles).

We can potentially avoid this issue by using the pandas nullable integer data type: https://pandas.pydata.org/docs/user_guide/integer_na.html.

import pyarrow

table = pyarrow.Table.from_pydict({"column": [0, None]})
df = table.to_pandas()
assert df.dtypes[0] == int, df.dtypes[0]

Traceback (most recent call last):
  File "/Users/balaji/Documents/GitHub/ray/1.py", line 5, in <module>
    assert df.dtypes[0] == int, df.dtypes[0]
           ^^^^^^^^^^^^^^^^^^^
AssertionError: float64

Component(s)

Python