Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large integers are truncated #61

Open
rossviljoen opened this issue Dec 18, 2024 · 2 comments
Open

Large integers are truncated #61

rossviljoen opened this issue Dec 18, 2024 · 2 comments

Comments

@rossviljoen
Copy link

rossviljoen commented Dec 18, 2024

Large Int32s in the plot data seem to get truncated, resulting in an unwanted loss of precision when plotting with PlutoPlot(Plot(...)), this is fixed when setting PlutoPlotly.floatval() = Val(false), so I think it's related to the preprocessing.

@rossviljoen rossviljoen changed the title Large floats are truncated Large integers are truncated Dec 18, 2024
@disberd
Copy link
Member

disberd commented Dec 18, 2024

Indeed by default all numbers are converted to Float32 for saving memory and speeding up transfer between JS and julia.
You can control this with PlutoPlotly.FORCE_FLOAT32[] = false but I guess overwriting the method achieves the same result.

I can probably skip converting Int32s into Float32 but not sure whether converting Int64 to Int32 or Float32. Any opinion on this @rossviljoen?

I also wanted this to be a setting that could be controlled on a plot-by-plot basis (hence the ScopedValue for the flag) but unfortunately the processing is done during the show method which is happening outside of the ScopedValue block, so at the moment it still needs to happen globally for all plots in a notebook...

@rossviljoen
Copy link
Author

I don't believe PlutoPlotly.FORCE_FLOAT32[] = false will work with a ScopedValue, as there's no setindex! method for it, hence why it's necessary to override the method. I'm not sure there's that much benefit to it being a ScopedValue anyway given the issue with show you mentioned.

I can probably skip converting Int32s into Float32 but not sure whether converting Int64 to Int32 or Float32

If it's necessary for performance, converting Int64 to Float32 likely makes more sense to avoid overflow issues. I think it would be good to document this behaviour (and how to disable it) in general though.

(PS. thanks for the excellent package!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants