Decimal Precision [Answered]

Feb 2, 2012 at 12:41am
I'm making a matrix class and comparing 2 matrices. So, I made an == method. However, it doesn't seem to work when I use doubles. I output my answers to the screen at 4 decimal points, so they look equal to me. But when I check the 2 values, I notice they differ by such a small amount, like .00000001 or so. Is there a way to truncate that so == works?
Last edited on Feb 2, 2012 at 1:00am
Feb 2, 2012 at 12:53am
You could work in integer space (multiply by 10n then round to integer and compare), or you could check that the difference between the two doubles is less than some small tolerance.
Feb 2, 2012 at 1:00am
Thank you. I can't believe I didn't think of checking for a small difference rather than relying on ==.
Topic archived. No new replies allowed.