Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unit Test Float Encode Exhaustively #11

Open
waylonflinn opened this issue Dec 16, 2015 · 2 comments
Open

Unit Test Float Encode Exhaustively #11

waylonflinn opened this issue Dec 16, 2015 · 2 comments

Comments

@waylonflinn
Copy link
Owner

The floating point encode step is a tricky bit to get right. Issue #10 shows that numerical stability issues can make a big difference in some edge cases.

Encoding each unique floating point value as an element of a 4098 square matrix should require only a few hundred of these matrices. This should be doable in a reasonable amount of time.

References

@waylonflinn waylonflinn added this to the 1.0 milestone Dec 16, 2015
@waylonflinn
Copy link
Owner Author

@robertleeplummerjr
Copy link

What is the current decimal precision? Is it better, for example than: https://github.com/gpujs/gpu.js/blob/develop/test/src/internal/precision.js#L7?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants