-
Notifications
You must be signed in to change notification settings - Fork 96
fix: use g17 format for doubles #434
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Conversation
It looks like the tests are currently failing due to some handling of the floating points:
This is a behavior change. Hopefully, someone does not depend on exact floating points, but I have concerns about this causing issues for existing users. I was hoping to have @bednar and @davidby-influx chime in on the change. Thanks! |
My suggestion is to change the tests to round the Something like: expected = 2.33333 got = ComputedValue() if Math.Round(got, 5) != expected This may not fix everything (because midpoint rounding can cause problems, but it will fix a lot of the cases. |
Thanks for the looking at the PR! The tests are primarily comparing two strings of line protocol. I think in those cases if we are good with updates to the floating points and it sounds like we are, I can update the expected. I'll let this this sit over night and see if Jakub chimes in. Thanks again, I appreciate the review. |
It looks good for me 👍 |
How will this work when we add a filter to compare
} |
Thank you both - the failing tests are updated, formatting corrected, and readme rebased. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
fixes: #408
Proposed Changes
Updates the format used for doubles when building a Point to use the G17 format. This ensures 17 bits of precision and that values are successful in round-trips. This is the recommendation from the dot.net docs as well.
Checklist
dotnet test
completes successfully