Decimal Precision [Answered]
I'm making a matrix class and comparing 2 matrices. So, I made an == method. However, it doesn't seem to work when I use doubles. I output my answers to the screen at 4 decimal points, so they look equal to me. But when I check the 2 values, I notice they differ by such a small amount, like .00000001 or so. Is there a way to truncate that so == works?
Last edited on
You could work in integer space (multiply by 10n then round to integer and compare), or you could check that the difference between the two doubles is less than some small tolerance.
Thank you. I can't believe I didn't think of checking for a small difference rather than relying on ==.
Topic archived. No new replies allowed.