Rounding

Apr 22, 2008 at 7:41pm
how would i round a decimal like .3333333333e103 to the 7th digit place.
like .66666666666666666666666666666666666666666666- to .6666667
Last edited on Apr 22, 2008 at 8:22pm
Apr 22, 2008 at 9:14pm
Here is some pseudocode:
1
2
3
4
Multiply the decimal value by 10eN where N is the number of digit places.
Add .5 * SIGN to the value, where SIGN is the sign (-1 or 1) of the value.
Truncate the decimal.
Divide the value by 10eN, where N is the number of digit places.


I used pseudocode because I didn't want to spoil your fun.

I hope this helps! ^_^

rpgfan
Apr 23, 2008 at 11:48pm
um i dont know how many digits it a calc
Apr 24, 2008 at 12:30am
If you are using floating point variables to store the numbers then rounding may not help you: there might not be enough precision, depending upon the type you're using and the decimal place you want to round to, to hold the rounded value....
Topic archived. No new replies allowed.