Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Profiles Int64 variables as float #1075

Open
anxti opened this issue Nov 27, 2023 · 1 comment
Open

Profiles Int64 variables as float #1075

anxti opened this issue Nov 27, 2023 · 1 comment
Assignees
Labels
Bug Something isn't working

Comments

@anxti
Copy link

anxti commented Nov 27, 2023

General Information:

  • OS: Win 10 Enterprise
  • Python version: 3.9.7
  • Library version: 0.10.7

Describe the bug:
Int64 columns in a dataframe are treated as float, "samples" lists float values, as well as all statistics do.
data_label = FLOAIs it possible that "Int64" (capital I) is not supported?
c

To Reproduce:

Expected behavior:
treat Int64 as Int64...

Screenshots:

Additional context:
Can't actually show too much details of the data but I'll try to explain...
I have selected data from a database into a python dataframe. Most of the DB fields are nullable, so they end up in python as float, because it supports NaN.
In order to use the correct data type, I have converted those fields to Int64 in python.
This yields for example:
Name: F*****, Length: 147190, dtype: Int64
For this variable the profile report says, "data_type": "string",
In 'statistics', "min": 1.0 , "max": 4.0, "sum": 117454.0, which looks like float.
The histogram bin_edges are all float, which I wouldn't expect for an int value.

"categorical": true, which it is for all int variables -> could be ok, but in fact in most of the cases these are not categories but simple measurement values.
'null_count': if this is supposed to count NaN-entries, it does not count correctly for Int64. Int64-NaN values seem to be denoted as (couldn't find out yet if there is a way to influence the NaN representation of Int64).
For the variable above the count is not available in the report (as is the case for all Int64 variables).

Hope this helps...

@anxti anxti added the Bug Something isn't working label Nov 27, 2023
@taylorfturner
Copy link
Contributor

Hey @anxti, thanks for opening an issue here.

Please fill out the rest of the issue template with a code snippet to reproduce / example data you are using -- that helps us work through issues a bit quicker. Might be slightly delayed in my responses this week as well but will do my best to keep an eye on any updates here. Best!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working
Projects
None yet
Development

No branches or pull requests

5 participants