Indigo Rose Software

Professional Software Development Tools

 
Results 1 to 2 of 2
  1. #1
    Join Date
    Jul 2003
    Posts
    49

    Simple floating-point subtraction

    I'm not sure the math is working here. Here are some examples:

    Result = 15.7 - 8.83 - 6.87. Result should be 0, but instead is 8.8817841970013e-016
    Result = 15.7 - 8.83 - 6.86. Result should be 0.01 but instead is 0.0099999999999989

    If the numbers are converted to integers before the math, and the result is converted back, the answer is correct.
    Is there something wrong here or should I always convert floating-point to integers before doing math?

    Thanks,
    Craig Booth

  2. #2
    Join Date
    Oct 2003
    Posts
    908
    That's sort of the nature of floating point math on computers. Take a look at this thread for some great examples and solutions on how to handle it:

    http://www.indigorose.com/forums/sho...light=rounding

Similar Threads

  1. Few simple questions...
    By daveinmb in forum AutoPlay Media Studio 5.0
    Replies: 7
    Last Post: 03-22-2004, 01:49 AM
  2. A simple database and printing??
    By patrick6 in forum AutoPlay Media Studio 5.0
    Replies: 34
    Last Post: 03-14-2004, 06:17 PM
  3. Simple web button getting errors
    By MWMTex in forum AutoPlay Media Studio 5.0
    Replies: 3
    Last Post: 02-03-2004, 07:18 PM
  4. Simple Question
    By joesoap in forum AutoPlay Menu Studio 3.0
    Replies: 1
    Last Post: 02-23-2001, 12:28 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts