PDA

View Full Version : Incorrect math?



dthompson16
08-28-2005, 08:47 PM
I am writing an app in which I'm trying to convert any # of seconds to minutes + seconds. When I tried to convert 55555.365 seconds, the result I got was 925 minutes and 55.3649999 seconds. The code I used is:
if Sec >= 60 then
NewNum = Sec / 60;
Min1 = Math.Floor(NewNum);
Expanded = Min1 * 60;
Sec1 = Sec - Expanded;
end
In 2 paragraph objects, I display Min1 and Sec1.
This works correctly if I enter 55.365, 555.365, or even 5555.365. However, entering 55555.365 gives the erroneous result. In the format XXXXX.365, I didn't encounter any problems until XXXXX got to be 32800 or higher, at which point the "9999" stuff started appearing. Ugh!
:huh

csd214
08-29-2005, 05:39 AM
Still I don’t quite understand how numbers are stored inside the computer but if you use the solution from your previous thread (http://www.indigorose.com/forums/showpost.php?p=56035&postcount=11) , just adding

Sec1 = Math.Round(Sec1, 10); -- round with the highest available precision
you’ll have the result 925 min 55.365 sec.

dthompson16
08-29-2005, 12:52 PM
Csd to the rescue once again! After midnite, I didn't recall that the newest release of AMS5 had the precision in the Math.Round function, but in the light of day, your msg hit me in the forehead with a bonk. Of course, it works like a charm. Thanks for the refresher!

csd214
08-29-2005, 01:15 PM
Just a pleasure! There are a lot of forum posts about Flash and Videos; stuff I’m not familiar with. I grasp the opportunity when I see an issue I “understand” (the decimal portion is a bit tricky). I have worked with figures and people, people and figures, all my life… :lol

Corey
08-29-2005, 02:27 PM
Csd to the rescue once again!
Nice one csd214. :)