Working with floats, adding precision
Pages: 12
Because, the 9 rounds the 4 to 5 giving 1.5 which rounds to 2. Right?
Thus technically 1.45+ should round to 2.
Thus technically 1.45+ should round to 2.
Last edited on
That's a slippery slope. With the same logic, 1.444445 should round up to 2. The fact is, rounding doesn't work like that.
You pick a number of digits you want after the comma, and look at the digit that's right below. If the digit is <5, truncate at that position. Otherwise, add 5 to the digit and truncate at that position.
Now, 1.49999 rounds with no decimal places to 1, but it rounds with 1 decimal place to 1.5, with 2 to 1.50, and with 4 to 1.4999.
You pick a number of digits you want after the comma, and look at the digit that's right below. If the digit is <5, truncate at that position. Otherwise, add 5 to the digit and truncate at that position.
Now, 1.49999 rounds with no decimal places to 1, but it rounds with 1 decimal place to 1.5, with 2 to 1.50, and with 4 to 1.4999.
That's correct. According to the "laws of math" for rounding, you start at the right side, and work left. (Not how a computer works)
Now I get what you are saying, (or I think I do) and I've been working with the formula that I made. Here's what I've come up with:
That there seems to work. Only doesn't when I enter 1.4444444444444445. :) But I don't think I'll have to worry about that.
Now I get what you are saying, (or I think I do) and I've been working with the formula that I made. Here's what I've come up with:
ceil(f)-f+.444445 <= 1That there seems to work. Only doesn't when I enter 1.4444444444444445. :) But I don't think I'll have to worry about that.
Last edited on
Topic archived. No new replies allowed.
Pages: 12