PDA

View Full Version : Simple floating-point subtraction



kraigyb
04-29-2004, 11:28 AM
I'm not sure the math is working here. Here are some examples:

Result = 15.7 - 8.83 - 6.87. Result should be 0, but instead is 8.8817841970013e-016
Result = 15.7 - 8.83 - 6.86. Result should be 0.01 but instead is 0.0099999999999989

If the numbers are converted to integers before the math, and the result is converted back, the answer is correct.
Is there something wrong here or should I always convert floating-point to integers before doing math?

Thanks,
Craig Booth

Ted Sullivan
04-29-2004, 11:53 AM
That's sort of the nature of floating point math on computers. Take a look at this thread for some great examples and solutions on how to handle it:

http://www.indigorose.com/forums/showthread.php?t=7285&highlight=rounding