You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Computing the mean and std of large frames (e.g. ~16 million points) can suffer from floating point error when they are performed using float32 values. Both mean and std can be passed dtype=np.float64 to perform these computations using float64 instead and thereby retain better accuracy. One place where this could be added is here:
Computing the mean and std of large frames (e.g. ~16 million points) can suffer from floating point error when they are performed using float32 values. Both
mean
andstd
can be passeddtype=np.float64
to perform these computations using float64 instead and thereby retain better accuracy. One place where this could be added is here:setigen/setigen/frame.py
Line 398 in 5ca049f
but perhaps it would be good to add a
dtype=np.float64
parameter to the enclosing function so that the user could specify a different type if desired.The text was updated successfully, but these errors were encountered: