Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] JIT GroupBy.apply corr() aggregation wrong results when one variable has zero variance #13875

Closed
brandon-b-miller opened this issue Aug 14, 2023 · 0 comments · Fixed by #13884
Assignees
Labels
bug Something isn't working numba Numba issue Python Affects Python cuDF API.

Comments

@brandon-b-miller
Copy link
Contributor

Describe the bug
When executing a corr aggregation in a groupby apply UDF, the result can diverge from pandas in some cases, specifically when one of the variables to be correlated has no variance.

Steps/Code to reproduce bug

import pandas as pd
import cudf

import numpy as np

df = pd.DataFrame(
    {
        'a':[0,0,0],
        'b':[1,1,1],
        'c':[2,2,2] 
    }
)
gdf = cudf.from_pandas(df)

def func(group):

    return group['b'].corr(group['c'])

pandas_result = df.groupby('a').apply(func)
cudf_result = gdf.groupby('a').apply(func, engine='jit')

print(pandas_result)
print(cudf_result)
a
0   NaN
dtype: float64
a
0    0.0
dtype: float64

Expected behavior
Results should be the same as pandas.

Environment overview (please complete the following information)
Bare-metal, 23.10

Additional Context
came up during #13813. This must be due to mishandling the denominator in corr.

@brandon-b-miller brandon-b-miller added bug Something isn't working numba Numba issue Python Affects Python cuDF API. labels Aug 14, 2023
@brandon-b-miller brandon-b-miller self-assigned this Aug 14, 2023
rapids-bot bot pushed a commit that referenced this issue Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working numba Numba issue Python Affects Python cuDF API.
Projects
Archived in project
1 participant